Service high availability standards (Pacemaker/Corosync)
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
Thesis of this document: HA without tested fencing is not HA, it is deferred data
corruption. A cluster that cannot kill a suspect node with certainty is not protecting the
service: it is waiting for the day two nodes write the same data at the same time.
The corollary most often forgotten: a badly operated cluster has worse availability than a
simple, well-monitored service. Every node, every agent, every constraint and every
fencing device is a new part that can fail — and it does fail. Complexity is the
enemy of uptime. Apply KISS before applying Pacemaker.
1. Scope and triggers
Applies to providing high availability to services running inside an operating system:
the decision of whether a cluster is needed at all, quorum and arbitration, fencing/STONITH, resource
management with Pacemaker, shared or replicated storage, patterns by service type,
cluster operation and failover exercises.
Triggers: pacemaker, pacemakerd, corosync, corosync-qnetd, corosync-qdevice,
votequorum, two_node, wait_for_all, last_man_standing, auto_tie_breaker, pcs,
pcsd, crm, crmsh, crm_mon, crm_resource, crm_simulate, crm_verify, cibadmin,
cib.xml, stonith-enabled, no-quorum-policy, resource-stickiness, migration-threshold,
failure-timeout, ocf:heartbeat:*, IPaddr2, Filesystem, systemd: as a resource class,
pcs constraint colocation|order, clone, promotable/master, fence_ipmilan,
fence_idrac, fence_ilo4, fence_apc_snmp, fence_sbd, fence_vmware_rest,
fence_aws/fence_gce, sbd, /etc/sysconfig/sbd, SBD_WATCHDOG_DEV, softdog,
drbdadm, drbd.conf, drbdsetup, dlm_controld, gfs2, mkfs.gfs2, ocfs2,
patroni.yml, patronictl, keepalived.conf, vrrp_instance, ha_cluster (system role),
cockpit-ha-cluster, "split-brain", "the cluster failed over on its own", "both nodes think
they are primary", "the resource does not start and I do not know why".
Internal arbitration rule: if the answer is written with pcs/crm/cibadmin over a
Pacemaker CIB, or in sbd.conf/drbd.conf, it belongs to this skill.
1.1 Boundary with proxmox-ve-standards (mirror of the rule already written there)
proxmox-ve-standards already set the arbitration rule for this pair, and this document mirrors it
verbatim:
If the resource that fails over is a PVE VM or container, it belongs to proxmox-ve-standards;
if it is a service inside an OS, it belongs here.
With two clarifications that must be clear:
- PVE ships its own HA stack —
pve-cluster/pmxcfs, pve-ha-manager, its HA groups and
its watchdog— and it is not operated with pcs or with crm. Running pcs on a PVE node is a
category error: there is no CIB to touch.
- Corosync is common to both. The principles of quorum, ring latency, link redundancy
and node arithmetic hold equally on both sides; what changes is the resource
manager on top.
1.2 HA is not DR (the domain's most common confusion)
|
HA (this skill) |
DR (bcdr-standards) |
| Responds to |
Failure of one component inside a failure domain |
Loss of the entire failure domain: room, data centre, region, provider |
| Horizon |
Seconds to minutes, automatic |
Minutes to days, with a human decision to declare the disaster |
| Mechanism |
Active redundancy and failover |
Recovery from a copy or an alternate site |
| What it does not protect against |
Logical deletion, corruption, ransomware, human error — it replicates them instantly |
(that is exactly what it exists for) |
| Metric |
Availability, MTTR |
RTO/RPO derived from business impact |
An HA cluster is not a backup and does not replace a continuity plan. Synchronous
replication is a DELETE propagated in milliseconds. And conversely: an impeccable DR plan does not
avoid Tuesday's minute of downtime. You need both, and they are different projects.
bcdr-standards sets RTO/RPO and the recovery order; this skill sets the mechanism that
supports the availability objective set by sre-practice-standards.
Not applicable: see proxmox-ve-standards (sister boundary, §1.1: HA of PVE VMs and containers
with ha-manager, HA groups, affinity rules and its watchdog),
kubernetes-standards (the orchestrator is the right answer to "make this service survive
its host going down": rescheduling, replicas, probes and PDBs. Pacemaker does not compete with that),
podman-systemd-containers-standards (cross-reference, no overlap: containers as systemd
services on a host. If the question is how they survive the host going down, the
honest answer is almost never with Pacemaker: an orchestrator, a load balancer in front of two
instances with the state outside, or downtime accepted in writing. Pacemaker managing
a host's containers is cluster complexity with none of its guarantees),
bcdr-standards (§1.2: RTO/RPO, BIA, DR exercises, alternate site, disaster
declaration), backup-recovery-standards (the copy and its tested restore — replication is not
a copy), sre-practice-standards (availability as an objective is theirs: SLI/SLO, error
budget, on-call, capacity; the mechanism to reach it belongs here. If the SLO is met without a
cluster, no cluster is built), incident-management-standards (declaration, roles and communication
of the incident; here the technical diagnosis of the cluster), data-platform-standards (the data
engine and its replication are theirs: PostgreSQL, its streaming replication, tuning, PITR, Redis,
Kafka; here the cluster mechanism that promotes or fails over — see §5.2, where the criterion is
that for PostgreSQL the default answer is not Pacemaker), linux-storage-standards
(LVM, multipath, iSCSI, local filesystems and LUKS underneath the cluster resource),
zfs-standards (the pool; ZFS is not a cluster filesystem and is not mounted on two nodes),
ceph-standards (frequent confusion: Ceph's monitor quorum is not Corosync's quorum,
and Ceph does not use STONITH — its high availability comes from RADOS through replication and the CRUSH map,
not from a resource manager; MON/OSD sizing and their failure domains are theirs),
linux-administration-standards (systemd: a cluster-managed resource is not
touched with systemctl — it is already forbidden there and confirmed here), networking-standards
(network design, VLANs, VRRP at the network level, MTU; here the cluster's use of the network and its
latency requirements), firewall-policy-standards (the filtering policy that must allow
Corosync traffic and the fencing devices),
observability-standards (design of the metrics and alerting stack; here what to watch on the
cluster and from where), identity-access-management-standards (BMC credentials and their
custody; here the fact that fencing needs those credentials and that they are highly privileged),
secrets-management-standards (where those credentials live), onprem-standards (umbrella:
its §1.3 sets the invariant HA without tested fencing is deferred corruption, which is this
document's thesis, and its §6 ceded the detail of the HA block to this skill),
microservices-architecture-standards (distributed resilience in the application —circuit
breakers, retries with backoff, bulkheads, sagas— versus infrastructure HA: a
well-designed application needs less cluster, and that is the right order of investment),
windows-server-ad-standards (WSFC and Windows clustering).
2. Default decisions / Toolchain
Verify the latest version on the web before pinning it in a real project (§8). In clusters,
the version that rules is the one your distribution packages and supports, not upstream's:
nothing is compiled by hand here.
| Component |
Status (Aug 2026) |
Criteria |
| Pacemaker |
3.0.3 (2026-07-28) and 2.1.11 (2026-06-29): both lines alive |
3.0 (GA 2025-01-08) exists to remove deprecated legacy syntax. Greenfield → 3.0.x if the distro ships it; otherwise 2.1.x, no drama |
| Corosync |
3.1.10 (2025-11-15) |
Annual cadence (releases in November). Stable and boring, which is what you want in the membership layer |
pcs |
0.12.3 (2026-07-10) / 0.11.12.1 (2026-07-14) — two maintained lines |
The tool on RHEL/Fedora/Ubuntu. On RHEL 10, the standalone pcsd web UI no longer exists: it moves to cockpit-ha-cluster |
crmsh |
5.0.0 (2025-08-15); 5.1.0-rc2 (2026-06-29) |
The tool on SUSE/SLES. SLE HA 16 is built around crm cluster init. Neither pcs nor crmsh is obsolete: you choose by vendor support, not by taste |
resource-agents |
4.18.0 (2026-04-08) |
Active |
fence-agents |
4.17.0 (2026-01-05) |
Active |
sbd |
1.5.2 (2023-01-09) — last release; repository with activity (commits in Jan 2026) |
Slow cadence: use it with the version your distro packages and do not assume recent upstream fixes. It is still the reference mechanism when there is no PDU/BMC |
| DRBD (module) |
9.3.3 (2026-07-01) out-of-tree from LINBIT; 10.0.0 in alpha — do not touch |
The DRBD in the mainline kernel is 8.4.11, from ~8 years ago. LINBIT is upstreaming 9.3 (it could land in Linux 7.2, Sep/Oct 2026). Today: out-of-tree module via DKMS, with everything that implies for kernel patching |
drbd-utils |
9.34.0 |
Do not mix drbd-utils 9.x with the in-tree 8.4 module: the config syntax does not match and it produces confusing failures |
| Patroni |
4.1.4 (2026-07-07); lines 4.0.10 and 3.3.11 patched the same day |
Default answer for PostgreSQL HA (§5.2) |
| keepalived |
2.4.3 |
VRRP: a lightweight alternative for a virtual IP without a full cluster (§5.5) |
| Automation |
The ha_cluster system role (RHEL) or crm cluster init (SUSE) |
A declarative, versioned cluster > an irreproducible interactive pcs session |
Status of cluster filesystems — an important, verified change:
- GFS2 is leaving Red Hat. The Resilient Storage Add-On is discontinued from RHEL
10 onwards: the
gfs2-utils, dlm and ctdb packages are discontinued and the gfs2 and dlm
modules have been removed from the RHEL 10 kernel. It remains supported on RHEL 7/8/9 until the end of their
maintenance cycle — that is your migration window, not an open-ended extension.
- OCFS2 is leaving SUSE. Deprecated in SLE HA 15 SP7 and it will not be in SLE HA 16; SUSE
documents migration to GFS2 (warning: GFS2 does not support reflink, unlike OCFS2).
- The two distributions have swapped their preferred cluster filesystem. Any
multi-vendor design has to take that on board.
- Consequence for criteria: on RHEL 10+ there is no supported shared-block filesystem
in the kernel. Active/active design over shared disk stops being a default
option; what remains is active/passive with XFS over shared storage, distributed
storage (CephFS and similar) or exporting over NFS/SMB from an active/passive pair. Choosing GFS2
today for something new requires justifying the lifecycle.
3. The prior question: do you really need HA?
It is the most important section in the document. Before installing anything, answer in writing:
- Which SPOF does the cluster remove, and which ones does it leave? A two-node cluster in the same rack,
with the same PDU, the same switch and the same storage array removes almost nothing: it moves the single
point of failure from the server to the switch. Draw the failure domains before buying
licences.
- What is the availability objective, and where does it come from? If it comes from an SLO with business
impact behind it (
sre-practice-standards), go ahead. If it comes from "we want it not to
go down", there is no requirement, there is anxiety.
- How much does a minute of downtime really cost? Compare it with the cost of operating a cluster:
a second node, fencing devices, a dedicated network, training, double patching windows and
a new failure mode nobody on the team has seen before.
- Who operates this at 3 in the morning? A cluster only one person understands reduces
real availability. If the on-call team cannot read a
crm_mon or know when to put the
cluster into maintenance, the cluster is a risk, not a control.
- Is the service's state compatible with failover? A service that takes 8 minutes
to recover its database after a dirty shutdown gains nothing from a 20-second failover.
3.1 Simpler alternatives that almost always suffice
They are ruled out in writing before building Pacemaker:
| Alternative |
When it suffices |
A service with Restart=on-failure and monitoring |
When the typical failure is the process dying, not the hardware burning. It covers most real incidents |
| A load balancer in front of N stateless backends |
The most robust pattern there is: no quorum, no fencing, no split-brain. If you can move the state out of the service, do it and save yourself the whole cluster |
| A replica with a documented manual promotion |
An RTO of minutes is acceptable. A tested runbook + a daily replica is worth more than a cluster nobody has exercised |
VRRP (keepalived) for the IP |
All you need is an IP moving between two nodes. No CIB, no fencing, no agents |
| An orchestrator |
The service is already in containers and what you want is automatic rescheduling |
| A managed service / the provider's cluster |
In the cloud: somebody else operates the HA and it comes out cheaper than yours |
Decision rule: build Pacemaker when you need automatic failover of a stateful
resource with guaranteed mutual exclusion — and you are willing to maintain tested fencing.
In any other case, there is a simpler answer, and the simpler answer is the correct one.
4. Quorum, split-brain and fencing
4.1 Quorum
- Three nodes as the minimum sane number. Quorum is arithmetic, not opinion: with 3 nodes
you survive the loss of 1 with a real majority.
- Two nodes with no arbiter is a data-corruption machine. Faced with a partition, both nodes
see "the other is not there" and both conclude the same thing. The only thing that stops both mounting
the data is fencing — and if fencing also depends on the partitioned network, it stops nothing.
votequorum parameters (verbatim text from the manual, so they are not quoted from memory):
two_node: "Enables two node cluster operations (default: 0)"; it artificially sets
quorum to 1 in two-node clusters. Note from the manual: "enabling two_node: 1 automatically
enables wait_for_all".
wait_for_all: "Enables Wait For All (WFA) feature (default: 0)" — the cluster
"will be quorate for the first time only after all nodes have been visible at least once at
the same time". Operational translation: it prevents a single node booting alone after a general
outage and declaring itself owner of everything. It is a real and cheap protection.
last_man_standing: dynamically recalculates expected_votes and quorum under certain
conditions, with a configurable window (10 s by default). Dangerous without impeccable
fencing: it lowers the quorum bar precisely when the cluster is degraded.
auto_tie_breaker: allows surviving the simultaneous loss of 50 % of the nodes
deterministically; by default the lowest nodeid wins, adjustable with
auto_tie_breaker_node.
corosync-qnetd + corosync-qdevice is the right answer for two nodes. A third
arbiter that does not have to be a cluster node (a small host in another failure
domain). A ridiculous cost compared with what it avoids. A two-node cluster with no qdevice and no
reliable fencing does not go into production.
- The arbiter has to be in another failure domain. A qnetd in the same rack, the same
switch or the same VM as a node arbitrates nothing.
- Corosync network: dedicated or at least isolated, with link redundancy (several
rings), low and stable latency. Corosync is sensitive to latency and jitter: a network
shared with backups or replication produces spurious fencings at 3 in the morning. The
design of that network belongs to networking-standards; the requirement comes from here.
no-quorum-policy: the sensible default is stop (stop resources on losing quorum).
ignore is the express route to split-brain and only makes sense in very specific configurations
with perfect fencing. no-quorum-policy=ignore copied from a blog is a common cause of
data loss.
4.2 Fencing / STONITH: the thesis
Without tested fencing there is no HA. The cluster cannot distinguish "the node is dead" from "the
node is not answering me but is still writing to disk". The only way to turn that
uncertainty into certainty is to kill the node and confirm it has died.
stonith-enabled=false is this domain's cardinal sin. It is the line that appears in
every "build your first cluster" tutorial and in every corruption postmortem.
It is forbidden outside a disposable lab, and in a lab it is documented as such.
A cluster with STONITH disabled is not a degraded cluster: it is two servers with a
system giving them permission to trample each other.
- Agents by platform (choose by what is underneath, not by what is easy):
| Substrate |
Agent |
Warning |
| Physical server with a BMC |
fence_ipmilan, fence_idrac, fence_ilo4/fence_ilo5 |
The BMC has to be on a network that survives the partition, and its credentials are extremely privileged (identity-access-management-standards). A BMC powered by the same PSU as the node is not always any use |
| Managed PDU |
fence_apc_snmp and equivalents |
Reliable and brutal. Careful with dual-power servers: you have to cut both feeds or nothing switches off |
| No BMC and no PDU |
SBD (fence_sbd) |
Disk mode (1–3 devices; 2–3 recommended for critical workloads) or diskless with a watchdog only |
| Hypervisor |
fence_vmware_rest, libvirt agents |
The hypervisor becomes the fencing SPOF: bear it in mind |
| Cloud |
fence_aws, fence_gce, fence_azure_arm |
They depend on the provider's API and on credentials: an outage of the provider's API is a fencing outage |
- SBD, verified and non-negotiable clarifications:
- Hardware watchdog, always. ClusterLabs is explicit: a software watchdog
depends on the OS working correctly, and is therefore not reliable for fencing.
softdog is fine for a lab and for nothing else.
- The
sbd daemon must be started before the cluster services, and the cluster
cannot manage it as a resource. It is infrastructure, not a resource.
- Disk mode: the shared device is a SPOF; with 2–3 devices in different arrays
it stops being one.
fencing loop: node A kills B, B boots, they do not agree, it kills A, and so on until
somebody stops it. It is mitigated with a delay on starting the cluster services after
boot and with wait_for_all.
fence racing: both nodes of a pair fire simultaneously and the cluster is left with
nobody. It is mitigated with an asymmetric delay on the fencing devices (pcmk_delay_base /
pcmk_delay_max, different per node) so one always wins.
- Fencing topology: when there are several mechanisms (BMC + PDU), the order and the
fallback are defined (
pcs stonith level), not two loose agents left competing.
4.3 THE MANDATORY TEST
Before a cluster takes production traffic, a node is powered off dirty. Not
pcs cluster stop, not reboot: cut the power, echo c > /proc/sysrq-trigger, or
unplug the active node's network. And you check:
- The surviving node fences the other and confirms it (it does not "believe" it has switched it off).
- The resources start on the survivor within the expected time, and it is timed.
- The data is intact: consistent filesystem, database with no corruption, no double
writes.
- The fenced node, on returning, does not take out the one that is serving.
- And it is repeated causing a network partition instead of the power-off: it is a different failure and
reveals whether fencing depends on the network that has just partitioned.
A cluster that has not passed this test is not in production, it is in testing — however
it behaves the rest of the time.
5. Pacemaker in practice and patterns by service
5.1 Resources, agents and constraints
- Agent class:
ocf: when an OCF agent exists (it has a real monitor and state
semantics), systemd: when what you manage is an existing, well-built unit. lsb:
only in legacy: init scripts lie about state.
- Every resource carries a
monitor operation with an explicit interval. A resource with no monitor is
a resource the cluster believes alive forever. And with a realistic timeout: a short
timeout on a slow service produces phantom failovers; a long one lengthens the real outage.
resource-stickiness > 0 by default. Without stickiness, the resource goes back to the
"preferred" node as soon as it reappears, causing a second free outage right after the
first. Failback is a decision, not a reflex.
migration-threshold + failure-timeout: they limit how many failures a resource tolerates on a
node before moving, and when the failure is forgotten. Without them, a resource bounces between nodes.
- Badly placed constraints are the number 1 cause of surprise failovers. Criteria:
- Colocation (
colocation) and ordering (order) are different things and are confused
daily: "together" does not imply "in this order". Both are declared when applicable.
- Intermediate
scores (neither INFINITY nor 0) produce behaviours nobody predicts.
Use INFINITY or do not add the constraint.
- A group (
group) is syntactic sugar for implicit colocation + ordering. Convenient and
readable; but it implies that a member's failure drags the following ones with it. If that is not
what you want, do not use a group.
clone for stateless active/active services; promotable for the
primary/replica pattern.
crm_simulate before applying any constraint change in production: it tells you
what the cluster is going to move before it moves it. It is the most underused tool in the
stack.
- The CIB is configuration, and it is versioned. It is exported (
pcs cluster cib / cibadmin -Q) to the
repository, and deployed with the ha_cluster role or equivalent (iac-standards). Editing
cib.xml by hand on disk is forbidden — use pcs/crm/cibadmin.
5.2 PostgreSQL: Patroni, not Pacemaker (a clear criterion)
**For PostgreSQL, the default answer today is Patroni + a distributed DCS (etcd or Consul)
- HAProxy in front. Pacemaker is the exception.**
Reasons, not tastes:
- Patroni understands PostgreSQL: replication roles, promotion, rewind, lag. Pacemaker is
a generic resource manager and everything it knows about Postgres comes from an agent.
- Patroni explicitly covers the ugly case: it demotes the primary when it is isolated from the
majority, and
maximum_lag_on_failover prevents promoting a replica that lags too far behind — which is
exactly the decision that ruins a badly executed failover.
Pacemaker is still the answer when:
- the database's HA is through shared storage, not through replication;
- PostgreSQL is one more resource inside a cluster that already manages other services and
OS dependencies;
- there is a vendor-supported stack built around Pacemaker (typical with SAP), and
stepping outside it leaves you without support.
Patroni traps to note in the design:
- The DCS is the new quorum. A single-node etcd, or —the classic— running on the same
machines as the database, turns a primary failure into a total outage. A distributed, quorate
etcd in different failure domains, or there is no HA.
- With only two PostgreSQL nodes quorum becomes awkward again; consider a third
node (even a witness) before accepting the design.
- Client routing (HAProxy against Patroni's REST API) is part of the design, not a
detail: without it, the failover happens and nobody notices.
The engine, its replication, its tuning and its PITR belong to data-platform-standards. Here only
which cluster mechanism is used and why is set.
5.3 Virtual IP
ocf:heartbeat:IPaddr2 is the most common resource and the easiest to break: the virtual IP is
colocated and ordered with the service that uses it (group or colocation+order), or you will end up with the
IP on one node and the service on another.
- Check that the gratuitous ARP arrives and that the switches update their table: a failover
that is technically correct with the network unaware of it is still an outage.
- If all you need is to move an IP, do not build Pacemaker:
keepalived (§5.5).
5.4 NFS / Samba in active-passive
- Pattern: shared storage +
Filesystem + virtual IP + the service, all in a
group, with strict ordering. Only one node mounts.
- What breaks people: the lock state. An NFS failover without migrating the lock
state leaves clients hanging or, worse, writing over locks that are no longer valid. The
state directory goes on the shared storage, and it is tested with real clients
writing during the failover, not with a
showmount.
- With Samba and CTDB in play: check the package's lifecycle in your distro before
designing (on RHEL 10,
ctdb is in the discontinued batch of the Resilient Storage Add-On).
5.5 VRRP with keepalived (the lightweight option, and often the right one)
- For HAProxy/nginx or another stateless frontend with a floating IP,
keepalived (2.4.x) does
the job with no CIB, no agents and no fencing.
- Its limits, stated plainly: VRRP has no quorum and no fencing. Faced with a partition,
both nodes can claim the IP (two MACs announcing the same address). It is acceptable
precisely because in front of stateless services a split-brain corrupts nothing:
it duplicates traffic, not data.
- Hard rule:
keepalived never in front of a stateful resource that cannot tolerate double
writing. There you need quorum and fencing, that is, Pacemaker (or the engine's own
mechanism, §5.2).
6. Shared and replicated storage
The rule that governs this whole section: a non-cluster filesystem, mounted on two
nodes at once, destroys the data. Not "may cause problems": it destroys it, and often in
silence, because each node has its own cache and its own journal. XFS, ext4, btrfs and ZFS
are not cluster filesystems. The only mechanism that prevents that double mount in the real
world is fencing (§4.2), and hence this document's thesis is what it is.
- Active/passive with XFS over shared storage is the default pattern and the one to
prefer: a single mount, a
Filesystem resource managed by the cluster, tested fencing.
Simple and sufficient in most cases.
- Active/active with GFS2 or OCFS2 requires a distributed lock manager (
dlm) and impeccable
fencing: without it, dlm cannot recover and the cluster hangs or corrupts. Besides,
their future is compromised in both big families (§2): GFS2 out of RHEL 10,
OCFS2 out of SLE HA 16. A new design on a cluster filesystem: justify the
lifecycle in writing, and plan the exit.
- DRBD when there is no shared array and you want block replication between nodes:
- Replication modes: A (asynchronous, RPO > 0), B (semi-synchronous) and C (synchronous,
RPO 0 and the one used in real HA). Choose C unless the link's latency makes it
impossible — and if it does, admit your RPO is not zero and note it in
bcdr-standards.
- Dual-primary is the classic trap. It only makes sense for the short, controlled case
(live VM migration) and requires a cluster filesystem on top. Historical LINBIT
guidance: in DRBD 9 there is "two-primary", not "multi-primary"; appearing to work is not
working. Permanent dual-primary for a normal filesystem = guaranteed corruption.
- Real operational debt: the module is out-of-tree (DKMS) and lags behind the kernel. Every
kernel update is a risk of booting with no
/dev/drbdX. Plan the patching
accounting for that. The effort to upstream DRBD 9.3 into mainline is under way but has not
landed yet; until then, DKMS.
- Do not mix
drbd-utils 9.x with the mainline kernel's 8.4 module: the configuration
syntax does not match.
- DRBD is not a backup. It replicates deletion and corruption at link speed.
7. Operation, gates and prohibitions
7.1 Operation
- Maintenance mode BEFORE touching anything.
pcs property set maintenance-mode=true (or
pcs node standby / crm node standby for one node) before patching, restarting a service,
testing something or looking with too much curiosity. The most common cause of a cluster incident
is an administrator operating the service by hand while the cluster watches.
- A cluster-managed resource is not touched with
systemctl. It is already forbidden in
linux-administration-standards; it is confirmed here: systemctl restart on a managed
resource makes the cluster see it as a failure and, depending on the configuration, a fencing.
- Leaving maintenance is part of the procedure, and you verify that the cluster sees the
real state (
crm_mon -1, no pending failures, a clean pcs status). A cluster that has spent
weeks in maintenance-mode is not providing HA and nobody has noticed.
- Rolling upgrades, node by node, with the cluster running:
- Pacemaker supports rolling upgrade from 2.0.0 onwards; from versions earlier than
2.0.0 it is not supported — you have to go through a 2.x release first.
- A node with Pacemaker 3.0+ does not connect to Pacemaker Remote nodes of 1.1.14 or earlier,
and Pacemaker 1 does not talk to Remote/bundles of 3.0+. Take inventory before starting.
- Pacemaker 3.0 validates the CIB strictly:
validate-with is mandatory, case-sensitive, and
does not accept old schemas (pacemaker-1.1, pacemaker-next, etc.). A CIB
that was working may fail to load after the upgrade. Test it with crm_verify
against the new schema before touching the first node.
- Monitoring of the cluster FROM OUTSIDE the cluster. A cluster that watches itself
reports perfectly right up until the moment it can no longer report. The "cluster without
quorum" or "node fenced" alert has to come out of a system that is not in the cluster
(
observability-standards). What to watch as a minimum:
- quorum present and the expected number of nodes;
stonith-enabled is true (alert if somebody disables it — it happens);
- resources stopped, failed or on an unexpected node;
- fencing failures (a fencing that fails is a high-severity incident, even if the
service stays up);
maintenance-mode active for more than X hours;
- Corosync retransmissions and token loss (an early symptom of the network that will cause the
next spurious fencing);
- the state of the fencing devices: the BMC/PDU is tested periodically, not when it is
needed;
- the health of SBD and the watchdog.
- The logs that matter:
pacemaker.log / the pacemaker and corosync journals, and above all
crm_mon --show-detail and the resource's failure history. In an incident, the question
is not "what happened" but "why did the cluster decide this", and that answer is in the
PE transition (crm_simulate over the transition file). Retention of those logs
sufficient for a postmortem (incident-management-standards).
- Fencing credentials: they are "switch this server off" credentials. They go in the secrets
manager (
secrets-management-standards), with rotation, and their management network segmented.
7.2 Exercises (recurring gates)
- Before production: the test of §4.3 — dirty power-off and network partition — with a
report: what was switched off, how long the failover took, data integrity, what failed.
- A scheduled failover at least every six months, in an agreed window, on the production
cluster. A failover only tested on installation day is not tested: the kernel, the agents,
the BMC firmware and the network rules have changed since then.
- Tested failback. The way back is one more failover and is usually less rehearsed than the
way out. It is exercised explicitly, and it is decided whether it is automatic (with low
stickiness) or manual (recommended by default).
- A report per exercise: date, scenario, measured failover time, deviation from
the objective, findings and actions with an owner. Without a report, the exercise does not count.
- Testing the fencing devices separately (
pcs stonith fence <node> in a window):
it confirms that the BMC answers, that the credentials are still valid and that the management
network reaches it. It is the component that degrades most silently.
crm_simulate in CI over the versioned CIB on any constraint change.
7.3 FORBIDDEN
- ❌
stonith-enabled=false in anything other than a disposable lab
documented as such. It is this document's number one prohibition.
- ❌ A cluster in production without the dirty power-off test of §4.3.
- ❌ A two-node cluster with no qdevice/qnetd (or without fencing that demonstrably resolves the
partition).
- ❌ An arbiter (qnetd) in the same failure domain as a cluster node.
- ❌
no-quorum-policy=ignore copied without understanding what it enables.
- ❌ A software watchdog (
softdog) as fencing in production: it is unreliable by design.
- ❌ Mounting a non-cluster filesystem (XFS, ext4, btrfs, ZFS) on two nodes at once.
- ❌ Permanent DRBD dual-primary under a filesystem that is not a cluster one.
- ❌ Confusing replication with backup, or HA with DR (§1.2).
- ❌ Operating a managed resource with
systemctl, or touching the service without maintenance-mode.
- ❌ Editing
cib.xml by hand on disk; configuring the cluster only through an interactive session without
leaving the CIB versioned in the repository.
- ❌ Constraints with intermediate
scores "to see what happens"; constraints without crm_simulate.
- ❌ A resource with no
monitor operation, or with a timeout copied from the example.
- ❌
resource-stickiness=0 (immediate automatic failback = a second free outage).
- ❌ Exposing the
pcsd daemon / its web interface to an untrusted network. The bulk of the recent
pcs CVEs comes from its dependencies packaged in pcsd (tornado, rack, lodash:
RHSA-2026:2452 / 2462 / 2469 / 2818 / 2819, Feb 2026), not from the cluster logic. Management
network, and patching at the pace of the distro's errata.
- ❌ Corosync over a network shared with backup or massive replication.
- ❌ Pacemaker managing a host's containers to "give them HA": see
podman-systemd-containers-standards. The answer is an orchestrator, a load balancer with the
state outside, or downtime accepted in writing.
- ❌
keepalived/VRRP in front of a stateful resource that cannot tolerate double writing.
- ❌ Building a cluster because "we want it not to go down", with no SLO, no failure domains
drawn and nobody on call who knows how to operate it.
- ❌ Compiling Pacemaker/Corosync by hand on a system with vendor support.
- ❌ A new design on GFS2/OCFS2 without justifying its lifecycle in writing (§2).
8. Mandatory web verification
Before pinning anything in a real project:
- The versions packaged by your distribution of
pacemaker, corosync, pcs/crmsh,
resource-agents, fence-agents and sbd — not upstream's. In clusters, vendor
support rules.
- The cluster filesystem's lifecycle in your distro: GFS2 discontinued from
RHEL 10 onwards (Resilient Storage Add-On; the
gfs2 and dlm modules out of the kernel), OCFS2
deprecated in SLE HA 15 SP7 and out of SLE HA 16. Confirm the exact end-of-support dates
for RHEL 9 before committing to a migration.
- Pacemaker 3.0 migration notes: strict CIB validation,
validate-with,
withdrawn schemas, Pacemaker Remote compatibility. Read ClusterLabs's 3.0 changes page
before the first rolling upgrade.
- The status of the specific fencing agent for your hardware/hypervisor/cloud and its parameters
(
pcmk_delay_base, pcmk_host_map, pcmk_reboot_action): they change between versions of
fence-agents.
- Security errata for
pcs/pcsd in your stream (the recent ones come from packaged
dependencies) and CVEs for corosync, pacemaker and sbd.
- Patroni: latest version and its compatibility matrix with the PostgreSQL version and with
the chosen DCS; and whether it is still the default recommendation over Pacemaker.
- DRBD: whether the 9.3 module has landed in the mainline kernel yet (LINBIT's estimate:
possibly Linux 7.2, Sep/Oct 2026) — it completely changes the maintenance equation
compared with DKMS.
- Corosync latency requirements and tuning (
token, consensus) for your topology, in
particular if there are links between rooms.
Declared gaps — do NOT fill from memory, verify before using:
- Date and content of the last
sbd release: the last verified tag is 1.5.2
(2023-01-09) with repository activity in January 2026, but it has not been confirmed whether
a later release exists nor which version RHEL 10 and SLE HA 16 package. Check it before
depending on a specific fix.
- The exact release date of
keepalived 2.4.3 (tag verified, date not obtained because of an
API rate limit) and its maintenance status.
- The date of
drbd-utils 9.34.0 (tag verified, date not obtained). And the exact relationship
between DRBD 9.3.3 and the 10.0.0 alpha branch: LINBIT's support plan has not been verified.
- The reason for the Patroni patch batch of 2026-07-07 (4.1.4, 4.0.10 and 3.3.11 on the same day,
the typical pattern of a security fix): not verified. Consult the advisory before
pinning a minimum version.
- The status and support of
pg_auto_failover as an alternative to Patroni in two-node
scenarios: mentioned in the sources but not verified as to current maintenance.
- Supported alternatives to GFS2 on RHEL 10 for active/active over shared block:
not verified beyond the observation that the add-on disappears. Confirm with Red
Hat before designing.
- The Corosync tuning parameters recommended today (
token, token_retransmits_before_loss_const,
consensus): not verified in this revision. Do not copy values from blogs.
- The exact compatibility of
crmsh 5.1 with Pacemaker 3.0.x (5.1.0 was at rc2 as of Jun 2026):
not verified.
If the web contradicts this document, the web wins — flag the discrepancy.
1---2name: ha-clustering-standards3description: High availability for services running inside an OS, with Pacemaker/Corosync as reference. Use when running pcs, crm/crmsh, crm_mon, crm_resource, crm_simulate, cibadmin, pcs stonith, pcs constraint or the ha_cluster system role, editing cib.xml, votequorum settings (two_node, wait_for_all, last_man_standing, auto_tie_breaker), corosync-qnetd/qdevice arbitration, STONITH agents (fence_ipmilan, fence_idrac, fence_ilo, fence_apc, fence_sbd, fence_vmware, fence_aws/fence_gce), sbd.conf and hardware watchdog fencing, stonith-enabled and no-quorum-policy, OCF resource agents, IPaddr2 virtual IPs, colocation and ordering constraints, resource-stickiness, clone and promotable resources, cluster maintenance-mode and standby, DRBD with drbdadm and drbd.conf, dlm with GFS2 or OCFS2 shared filesystems, Patroni patroni.yml versus a Pacemaker-managed PostgreSQL, keepalived.conf VRRP as a lighter alternative, split-brain and fence racing incidents, or answering whether a service needs a cluster at all.4---56# Service high availability standards (Pacemaker/Corosync)78Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).910> **Thesis of this document**: **HA without tested fencing is not HA, it is deferred data11> corruption.** A cluster that cannot kill a suspect node with certainty is not protecting the12> service: it is waiting for the day two nodes write the same data at the same time.13>14> **The corollary most often forgotten**: **a badly operated cluster has worse availability than a15> simple, well-monitored service.** Every node, every agent, every constraint and every16> fencing device is a new part that can fail — and it does fail. Complexity is the17> enemy of *uptime*. Apply KISS before applying Pacemaker.1819## 1. Scope and triggers2021Applies to providing high availability to **services running inside an operating system**:22the decision of whether a cluster is needed at all, quorum and arbitration, fencing/STONITH, resource23management with Pacemaker, shared or replicated storage, patterns by service type,24cluster operation and failover exercises.2526Triggers: `pacemaker`, `pacemakerd`, `corosync`, `corosync-qnetd`, `corosync-qdevice`,27`votequorum`, `two_node`, `wait_for_all`, `last_man_standing`, `auto_tie_breaker`, `pcs`,28`pcsd`, `crm`, `crmsh`, `crm_mon`, `crm_resource`, `crm_simulate`, `crm_verify`, `cibadmin`,29`cib.xml`, `stonith-enabled`, `no-quorum-policy`, `resource-stickiness`, `migration-threshold`,30`failure-timeout`, `ocf:heartbeat:*`, `IPaddr2`, `Filesystem`, `systemd:` as a resource class,31`pcs constraint colocation|order`, `clone`, `promotable`/`master`, `fence_ipmilan`,32`fence_idrac`, `fence_ilo4`, `fence_apc_snmp`, `fence_sbd`, `fence_vmware_rest`,33`fence_aws`/`fence_gce`, `sbd`, `/etc/sysconfig/sbd`, `SBD_WATCHDOG_DEV`, `softdog`,34`drbdadm`, `drbd.conf`, `drbdsetup`, `dlm_controld`, `gfs2`, `mkfs.gfs2`, `ocfs2`,35`patroni.yml`, `patronictl`, `keepalived.conf`, `vrrp_instance`, `ha_cluster` (system role),36`cockpit-ha-cluster`, "split-brain", "the cluster failed over on its own", "both nodes think37they are primary", "the resource does not start and I do not know why".3839**Internal arbitration rule**: if the answer is written with `pcs`/`crm`/`cibadmin` over a40Pacemaker CIB, or in `sbd.conf`/`drbd.conf`, it belongs to this skill.4142### 1.1 Boundary with `proxmox-ve-standards` (mirror of the rule already written there)4344`proxmox-ve-standards` already set the arbitration rule for this pair, and **this document mirrors it45verbatim**:4647> **If the resource that fails over is a PVE VM or container, it belongs to `proxmox-ve-standards`;48> if it is a service inside an OS, it belongs here.**4950With two clarifications that must be clear:5152- **PVE ships its own HA stack** —`pve-cluster`/`pmxcfs`, `pve-ha-manager`, its HA groups and53 its watchdog— and **it is not operated with `pcs` or with `crm`**. Running `pcs` on a PVE node is a54 category error: there is no CIB to touch.55- **Corosync is common to both.** The principles of quorum, ring latency, link redundancy56 and node arithmetic hold equally on both sides; what changes is the resource57 manager on top.5859### 1.2 HA is not DR (the domain's most common confusion)6061| | HA (this skill) | DR (`bcdr-standards`) |62|---|---|---|63| Responds to | Failure of **one component** inside a failure domain | Loss of the **entire failure domain**: room, data centre, region, provider |64| Horizon | Seconds to minutes, **automatic** | Minutes to days, **with a human decision** to declare the disaster |65| Mechanism | Active redundancy and failover | Recovery from a copy or an alternate site |66| What it does **not** protect against | Logical deletion, corruption, ransomware, human error — **it replicates them instantly** | (that is exactly what it exists for) |67| Metric | Availability, MTTR | **RTO/RPO** derived from business impact |6869**An HA cluster is not a backup and does not replace a continuity plan.** Synchronous70replication is a `DELETE` propagated in milliseconds. And conversely: an impeccable DR plan does not71avoid Tuesday's minute of downtime. **You need both, and they are different projects.**72`bcdr-standards` sets RTO/RPO and the recovery order; this skill sets the mechanism that73supports the availability objective set by `sre-practice-standards`.7475**Not applicable**: see `proxmox-ve-standards` (**sister boundary, §1.1**: HA of PVE VMs and containers76with `ha-manager`, HA groups, affinity rules and its watchdog),77`kubernetes-standards` (**the orchestrator is the right answer to "make this service survive78its host going down"**: rescheduling, replicas, probes and PDBs. Pacemaker does not compete with that),79`podman-systemd-containers-standards` (**cross-reference, no overlap**: containers as systemd80services on a host. If the question is how they survive the host going down, the81honest answer is **almost never with Pacemaker**: an orchestrator, a load balancer in front of two82instances with the state outside, or downtime accepted in writing. Pacemaker managing83a host's containers is cluster complexity with none of its guarantees),84`bcdr-standards` (**§1.2**: RTO/RPO, BIA, DR exercises, alternate site, disaster85declaration), `backup-recovery-standards` (the copy and its tested restore — **replication is not86a copy**), `sre-practice-standards` (**availability as an objective is theirs**: SLI/SLO, error87budget, on-call, capacity; **the mechanism to reach it belongs here**. If the SLO is met without a88cluster, no cluster is built), `incident-management-standards` (declaration, roles and communication89of the incident; here the technical diagnosis of the cluster), `data-platform-standards` (**the data90engine and its replication are theirs**: PostgreSQL, its streaming replication, tuning, PITR, Redis,91Kafka; **here the cluster mechanism that promotes or fails over** — see §5.2, where the criterion is92that for PostgreSQL the default answer is **not** Pacemaker), `linux-storage-standards`93(LVM, multipath, iSCSI, local filesystems and LUKS **underneath** the cluster resource),94`zfs-standards` (the pool; ZFS **is not** a cluster filesystem and is not mounted on two nodes),95`ceph-standards` (**frequent confusion: Ceph's monitor quorum is not Corosync's quorum,96and Ceph does not use STONITH** — its high availability comes from RADOS through replication and the CRUSH map,97not from a resource manager; MON/OSD sizing and their failure domains are theirs),98`linux-administration-standards` (**systemd**: a cluster-managed resource **is not99touched with `systemctl`** — it is already forbidden there and confirmed here), `networking-standards`100(network design, VLANs, VRRP at the network level, MTU; here the cluster's use of the network and its101latency requirements), `firewall-policy-standards` (the filtering policy that must allow102Corosync traffic and the fencing devices),103`observability-standards` (design of the metrics and alerting stack; here **what** to watch on the104cluster and **from where**), `identity-access-management-standards` (BMC credentials and their105custody; here the fact that fencing needs those credentials and that they are highly privileged),106`secrets-management-standards` (where those credentials live), `onprem-standards` (**umbrella**:107its §1.3 sets the invariant *HA without tested fencing is deferred corruption*, which is this108document's thesis, and its §6 ceded the detail of the HA block to this skill),109`microservices-architecture-standards` (**distributed resilience in the application** —circuit110breakers, retries with backoff, bulkheads, sagas— versus infrastructure HA: **a111well-designed application needs less cluster**, and that is the right order of investment),112`windows-server-ad-standards` (WSFC and Windows clustering).113114## 2. Default decisions / Toolchain115116> Verify the latest version on the web before pinning it in a real project (§8). In clusters,117> **the version that rules is the one your distribution packages and supports**, not upstream's:118> nothing is compiled by hand here.119120| Component | Status (Aug 2026) | Criteria |121|---|---|---|122| Pacemaker | **3.0.3** (2026-07-28) and **2.1.11** (2026-06-29): **both lines alive** | 3.0 (GA 2025-01-08) exists to **remove deprecated legacy syntax**. Greenfield → 3.0.x if the distro ships it; otherwise 2.1.x, no drama |123| Corosync | **3.1.10** (2025-11-15) | Annual cadence (releases in November). Stable and boring, which is what you want in the membership layer |124| `pcs` | **0.12.3** (2026-07-10) / **0.11.12.1** (2026-07-14) — two maintained lines | **The tool on RHEL/Fedora/Ubuntu.** On RHEL 10, the **standalone `pcsd` web UI no longer exists**: it moves to `cockpit-ha-cluster` |125| `crmsh` | **5.0.0** (2025-08-15); **5.1.0-rc2** (2026-06-29) | **The tool on SUSE/SLES.** SLE HA 16 is built around `crm cluster init`. **Neither `pcs` nor `crmsh` is obsolete**: you choose by vendor support, not by taste |126| `resource-agents` | **4.18.0** (2026-04-08) | Active |127| `fence-agents` | **4.17.0** (2026-01-05) | Active |128| `sbd` | **1.5.2** (2023-01-09) — last *release*; repository with activity (commits in Jan 2026) | **Slow cadence**: use it with the version your distro packages and do not assume recent upstream fixes. It is still the reference mechanism when there is no PDU/BMC |129| DRBD (module) | **9.3.3** (2026-07-01) out-of-tree from LINBIT; **10.0.0 in alpha** — do not touch | **The DRBD in the mainline kernel is 8.4.11, from ~8 years ago.** LINBIT is upstreaming 9.3 (it could land in Linux 7.2, Sep/Oct 2026). Today: **out-of-tree module via DKMS**, with everything that implies for kernel patching |130| `drbd-utils` | **9.34.0** | **Do not mix `drbd-utils` 9.x with the in-tree 8.4 module**: the config syntax does not match and it produces confusing failures |131| Patroni | **4.1.4** (2026-07-07); lines 4.0.10 and 3.3.11 patched the same day | **Default answer for PostgreSQL HA** (§5.2) |132| keepalived | **2.4.3** | VRRP: a **lightweight** alternative for a virtual IP without a full cluster (§5.5) |133| Automation | The **`ha_cluster`** system role (RHEL) or `crm cluster init` (SUSE) | A declarative, versioned cluster > an irreproducible interactive `pcs` session |134135**Status of cluster filesystems — an important, verified change**:136137- **GFS2 is leaving Red Hat.** The *Resilient Storage Add-On* **is discontinued from RHEL138 10 onwards**: the `gfs2-utils`, `dlm` and `ctdb` packages are discontinued and **the `gfs2` and `dlm`139 modules have been removed from the RHEL 10 kernel**. It remains supported on RHEL 7/8/9 until the end of their140 maintenance cycle — that is your migration window, not an open-ended extension.141- **OCFS2 is leaving SUSE.** Deprecated in SLE HA 15 SP7 and **it will not be in SLE HA 16**; SUSE142 documents migration **to GFS2** (warning: GFS2 **does not support reflink**, unlike OCFS2).143- **The two distributions have swapped their preferred cluster filesystem.** Any144 multi-vendor design has to take that on board.145- **Consequence for criteria**: on RHEL 10+ **there is no supported shared-block filesystem146 in the kernel**. Active/active design over shared disk stops being a default147 option; what remains is active/passive with XFS over shared storage, distributed148 storage (CephFS and similar) or exporting over NFS/SMB from an active/passive pair. **Choosing GFS2149 today for something new requires justifying the lifecycle.**150151## 3. The prior question: do you really need HA?152153**It is the most important section in the document.** Before installing anything, answer in writing:1541551. **Which SPOF does the cluster remove, and which ones does it leave?** A two-node cluster in the same rack,156 with the same PDU, the same switch and the same storage array removes almost nothing: it moves the single157 point of failure from the server to the switch. **Draw the failure domains before buying158 licences.**1592. **What is the availability objective, and where does it come from?** If it comes from an SLO with business160 impact behind it (`sre-practice-standards`), go ahead. If it comes from "we want it not to161 go down", there is no requirement, there is anxiety.1623. **How much does a minute of downtime really cost?** Compare it with the cost of operating a cluster:163 a second node, fencing devices, a dedicated network, training, double patching windows and164 a new failure mode nobody on the team has seen before.1654. **Who operates this at 3 in the morning?** A cluster only one person understands **reduces**166 real availability. If the on-call team cannot read a `crm_mon` or know when to put the167 cluster into maintenance, the cluster is a risk, not a control.1685. **Is the service's state compatible with failover?** A service that takes 8 minutes169 to recover its database after a dirty shutdown gains nothing from a 20-second failover.170171### 3.1 Simpler alternatives that almost always suffice172173They are ruled out **in writing** before building Pacemaker:174175| Alternative | When it suffices |176|---|---|177| **A service with `Restart=on-failure` and monitoring** | When the typical failure is the process dying, not the hardware burning. It covers most real incidents |178| **A load balancer in front of N stateless backends** | The most robust pattern there is: no quorum, no fencing, no split-brain. **If you can move the state out of the service, do it and save yourself the whole cluster** |179| **A replica with a documented manual promotion** | An RTO of minutes is acceptable. A tested runbook + a daily replica is worth more than a cluster nobody has exercised |180| **VRRP (`keepalived`) for the IP** | All you need is an IP moving between two nodes. No CIB, no fencing, no agents |181| **An orchestrator** | The service is already in containers and what you want is automatic rescheduling |182| **A managed service / the provider's cluster** | In the cloud: somebody else operates the HA and it comes out cheaper than yours |183184**Decision rule**: build Pacemaker when you need **automatic failover of a stateful185resource with guaranteed mutual exclusion** — and you are willing to maintain tested fencing.186In any other case, there is a simpler answer, and the simpler answer is the correct one.187188## 4. Quorum, split-brain and fencing189190### 4.1 Quorum191192- **Three nodes as the minimum sane number.** Quorum is arithmetic, not opinion: with 3 nodes193 you survive the loss of 1 with a real majority.194- **Two nodes with no arbiter is a data-corruption machine.** Faced with a partition, both nodes195 see "the other is not there" and both conclude the same thing. The only thing that stops both mounting196 the data is fencing — and if fencing also depends on the partitioned network, it stops nothing.197- `votequorum` parameters (verbatim text from the manual, so they are not quoted from memory):198 - **`two_node`**: *"Enables two node cluster operations (default: 0)"*; it artificially sets199 quorum to 1 in two-node clusters. Note from the manual: *"enabling `two_node: 1` automatically200 enables `wait_for_all`"*.201 - **`wait_for_all`**: *"Enables Wait For All (WFA) feature (default: 0)"* — the cluster202 *"will be quorate for the first time only after all nodes have been visible at least once at203 the same time"*. Operational translation: **it prevents a single node booting alone after a general204 outage and declaring itself owner of everything**. It is a real and cheap protection.205 - **`last_man_standing`**: dynamically recalculates `expected_votes` and quorum under certain206 conditions, with a configurable window (10 s by default). **Dangerous without impeccable207 fencing**: it lowers the quorum bar precisely when the cluster is degraded.208 - **`auto_tie_breaker`**: allows surviving the simultaneous loss of 50 % of the nodes209 deterministically; by default the lowest `nodeid` wins, adjustable with210 `auto_tie_breaker_node`.211- **`corosync-qnetd` + `corosync-qdevice` is the right answer for two nodes.** A third212 arbiter that **does not have to be a cluster node** (a small host in another failure213 domain). A ridiculous cost compared with what it avoids. **A two-node cluster with no qdevice and no214 reliable fencing does not go into production.**215- **The arbiter has to be in another failure domain.** A qnetd in the same rack, the same216 switch or the same VM as a node arbitrates nothing.217- **Corosync network**: dedicated or at least isolated, with **link redundancy** (several218 `ring`s), low and stable latency. Corosync is sensitive to latency and jitter: **a network219 shared with backups or replication produces spurious fencings at 3 in the morning**. The220 design of that network belongs to `networking-standards`; the requirement comes from here.221- **`no-quorum-policy`**: the sensible default is `stop` (stop resources on losing quorum).222 `ignore` is the express route to split-brain and only makes sense in very specific configurations223 with perfect fencing. **`no-quorum-policy=ignore` copied from a blog is a common cause of224 data loss.**225226### 4.2 Fencing / STONITH: the thesis227228**Without tested fencing there is no HA.** The cluster cannot distinguish "the node is dead" from "the229node is not answering me but is still writing to disk". The only way to turn that230uncertainty into certainty is **to kill the node** and confirm it has died.231232- **`stonith-enabled=false` is this domain's cardinal sin.** It is the line that appears in233 every "build your first cluster" tutorial and in every corruption postmortem.234 It is **forbidden** outside a disposable lab, and in a lab it is documented as such.235 A cluster with STONITH disabled is not a degraded cluster: it is two servers with a236 system giving them permission to trample each other.237- **Agents by platform** (choose by what is underneath, not by what is easy):238239| Substrate | Agent | Warning |240|---|---|---|241| Physical server with a BMC | `fence_ipmilan`, `fence_idrac`, `fence_ilo4`/`fence_ilo5` | **The BMC has to be on a network that survives the partition**, and its credentials are extremely privileged (`identity-access-management-standards`). A BMC powered by the same PSU as the node is not always any use |242| Managed PDU | `fence_apc_snmp` and equivalents | Reliable and brutal. Careful with dual-power servers: **you have to cut both feeds** or nothing switches off |243| No BMC and no PDU | **SBD** (`fence_sbd`) | Disk mode (1–3 devices; 2–3 recommended for critical workloads) or **diskless** with a watchdog only |244| Hypervisor | `fence_vmware_rest`, libvirt agents | The hypervisor becomes the fencing SPOF: bear it in mind |245| Cloud | `fence_aws`, `fence_gce`, `fence_azure_arm` | They depend on the provider's API and on credentials: an outage of the provider's API is a fencing outage |246247- **SBD**, verified and non-negotiable clarifications:248 - **Hardware watchdog, always.** ClusterLabs is explicit: a **software** watchdog249 depends on the OS working correctly, and is therefore **not reliable for fencing**.250 `softdog` is fine for a lab and **for nothing else**.251 - The `sbd` daemon **must be started before the cluster services**, and **the cluster252 cannot manage it as a resource**. It is infrastructure, not a resource.253 - Disk mode: the shared device is a SPOF; with 2–3 devices in different arrays254 it stops being one.255- **`fencing loop`**: node A kills B, B boots, they do not agree, it kills A, and so on until256 somebody stops it. It is mitigated with a delay on starting the cluster services after257 boot and with `wait_for_all`.258- **`fence racing`**: both nodes of a pair fire simultaneously and the cluster is left with259 nobody. It is mitigated with an **asymmetric delay** on the fencing devices (`pcmk_delay_base` /260 `pcmk_delay_max`, different per node) so one always wins.261- **Fencing topology**: when there are several mechanisms (BMC + PDU), the **order and the262 fallback** are defined (`pcs stonith level`), not two loose agents left competing.263264### 4.3 THE MANDATORY TEST265266**Before a cluster takes production traffic, a node is powered off dirty.** Not267`pcs cluster stop`, not `reboot`: **cut the power, `echo c > /proc/sysrq-trigger`, or268unplug the active node's network.** And you check:2692701. The surviving node **fences** the other and confirms it (it does not "believe" it has switched it off).2712. The resources start on the survivor within the expected time, and it is **timed**.2723. The data is **intact**: consistent filesystem, database with no corruption, no double273 writes.2744. The fenced node, on returning, **does not take out** the one that is serving.2755. And it is repeated **causing a network partition** instead of the power-off: it is a different failure and276 reveals whether fencing depends on the network that has just partitioned.277278**A cluster that has not passed this test is not in production, it is in testing** — however279it behaves the rest of the time.280281## 5. Pacemaker in practice and patterns by service282283### 5.1 Resources, agents and constraints284285- **Agent class**: `ocf:` when an OCF agent exists (it has a real `monitor` and state286 semantics), `systemd:` when what you manage is an existing, well-built unit. **`lsb:`287 only in legacy**: init scripts lie about state.288- **Every resource carries a `monitor` operation with an explicit `interval`.** A resource with no monitor is289 a resource the cluster believes alive forever. And with a **realistic** `timeout`: a short290 timeout on a slow service produces phantom failovers; a long one lengthens the real outage.291- **`resource-stickiness` > 0 by default.** Without *stickiness*, the resource goes back to the292 "preferred" node as soon as it reappears, causing **a second free outage** right after the293 first. Failback is a decision, not a reflex.294- **`migration-threshold` + `failure-timeout`**: they limit how many failures a resource tolerates on a295 node before moving, and when the failure is forgotten. Without them, a resource bounces between nodes.296- **Badly placed constraints are the number 1 cause of surprise failovers.** Criteria:297 - **Colocation** (`colocation`) and **ordering** (`order`) are different things and are confused298 daily: "together" does not imply "in this order". **Both** are declared when applicable.299 - Intermediate `score`s (neither `INFINITY` nor 0) produce behaviours nobody predicts.300 **Use `INFINITY` or do not add the constraint.**301 - A **group** (`group`) is syntactic sugar for implicit colocation + ordering. Convenient and302 readable; but it implies that **a member's failure drags the following ones with it**. If that is not303 what you want, do not use a group.304 - **`clone`** for stateless active/active services; **`promotable`** for the305 primary/replica pattern.306 - **`crm_simulate` before applying** any constraint change in production: it tells you307 what the cluster is going to move **before** it moves it. It is the most underused tool in the308 stack.309- **The CIB is configuration, and it is versioned.** It is exported (`pcs cluster cib` / `cibadmin -Q`) to the310 repository, and deployed with the `ha_cluster` role or equivalent (`iac-standards`). **Editing311 `cib.xml` by hand on disk is forbidden** — use `pcs`/`crm`/`cibadmin`.312313### 5.2 PostgreSQL: **Patroni, not Pacemaker** (a clear criterion)314315**For PostgreSQL, the default answer today is Patroni + a distributed DCS (etcd or Consul)316+ HAProxy in front. Pacemaker is the exception.**317318Reasons, not tastes:319- Patroni **understands PostgreSQL**: replication roles, promotion, *rewind*, lag. Pacemaker is320 a generic resource manager and everything it knows about Postgres comes from an agent.321- Patroni explicitly covers the ugly case: **it demotes the primary when it is isolated from the322 majority**, and `maximum_lag_on_failover` prevents promoting a replica that lags too far behind — which is323 exactly the decision that ruins a badly executed failover.324325**Pacemaker is still the answer when**:326- the database's HA is **through shared storage**, not through replication;327- PostgreSQL is **one more resource** inside a cluster that already manages other services and328 OS dependencies;329- there is a vendor-supported stack built around Pacemaker (typical with SAP), and330 stepping outside it leaves you without support.331332**Patroni traps to note in the design**:333- **The DCS is the new quorum.** A single-node etcd, or —the classic— running on the same334 machines as the database, turns a primary failure into a total outage. **A distributed, quorate335 etcd in different failure domains**, or there is no HA.336- With **only two PostgreSQL nodes** quorum becomes awkward again; consider a third337 node (even a witness) before accepting the design.338- Client routing (HAProxy against Patroni's REST API) is **part of the design**, not a339 detail: without it, the failover happens and nobody notices.340341**The engine, its replication, its tuning and its PITR belong to `data-platform-standards`.** Here only342**which cluster mechanism is used and why** is set.343344### 5.3 Virtual IP345346- `ocf:heartbeat:IPaddr2` is the most common resource and the easiest to break: **the virtual IP is347 colocated and ordered with the service that uses it** (group or colocation+order), or you will end up with the348 IP on one node and the service on another.349- Check that the gratuitous ARP arrives and that the switches update their table: a failover350 that is technically correct with the network unaware of it is still an outage.351- If **all** you need is to move an IP, **do not build Pacemaker**: `keepalived` (§5.5).352353### 5.4 NFS / Samba in active-passive354355- Pattern: shared storage + `Filesystem` + virtual IP + the service, all in a356 **group**, with strict ordering. Only one node mounts.357- **What breaks people**: the **lock state**. An NFS failover without migrating the lock358 state leaves clients hanging or, worse, writing over locks that are no longer valid. The359 state directory goes on the shared storage, and **it is tested with real clients360 writing during the failover**, not with a `showmount`.361- With Samba and CTDB in play: check the package's lifecycle in your distro before362 designing (on RHEL 10, `ctdb` is in the discontinued batch of the Resilient Storage Add-On).363364### 5.5 VRRP with `keepalived` (the lightweight option, and often the right one)365366- For **HAProxy/nginx or another stateless frontend** with a floating IP, `keepalived` (2.4.x) does367 the job with no CIB, no agents and no fencing.368- **Its limits, stated plainly**: VRRP **has no quorum and no fencing**. Faced with a partition,369 both nodes can claim the IP (two MACs announcing the same address). It is acceptable370 precisely because **in front of stateless services a split-brain corrupts nothing**:371 it duplicates traffic, not data.372- **Hard rule**: `keepalived` **never** in front of a stateful resource that cannot tolerate double373 writing. There you need quorum and fencing, that is, Pacemaker (or the engine's own374 mechanism, §5.2).375376## 6. Shared and replicated storage377378**The rule that governs this whole section**: **a non-cluster filesystem, mounted on two379nodes at once, destroys the data.** Not "may cause problems": it destroys it, and often in380silence, because each node has its own cache and its own journal. XFS, ext4, btrfs and ZFS381**are not cluster filesystems**. The only mechanism that prevents that double mount in the real382world is fencing (§4.2), and hence this document's thesis is what it is.383384- **Active/passive with XFS over shared storage** is the default pattern and the one to385 prefer: a single mount, a `Filesystem` resource managed by the cluster, tested fencing.386 Simple and sufficient in most cases.387- **Active/active with GFS2 or OCFS2** requires a distributed lock manager (`dlm`) and **impeccable388 fencing**: without it, `dlm` cannot recover and the cluster hangs or corrupts. Besides,389 **their future is compromised in both big families** (§2): GFS2 out of RHEL 10,390 OCFS2 out of SLE HA 16. **A new design on a cluster filesystem: justify the391 lifecycle in writing, and plan the exit.**392- **DRBD** when there is no shared array and you want block replication between nodes:393 - Replication modes: **A** (asynchronous, RPO > 0), **B** (semi-synchronous) and **C** (synchronous,394 RPO 0 and the one used in real HA). **Choose C unless the link's latency makes it395 impossible** — and if it does, admit your RPO is not zero and note it in `bcdr-standards`.396 - **Dual-primary is the classic trap.** It only makes sense for the short, controlled case397 (live VM migration) and **requires a cluster filesystem on top**. Historical LINBIT398 guidance: in DRBD 9 there is "two-primary", not "multi-primary"; appearing to work is not399 working. **Permanent dual-primary for a normal filesystem = guaranteed corruption.**400 - **Real operational debt**: the module is **out-of-tree** (DKMS) and lags behind the kernel. Every401 kernel update is a risk of booting with no `/dev/drbdX`. Plan the patching402 accounting for that. The effort to upstream DRBD 9.3 into mainline is under way but **has not403 landed yet**; until then, DKMS.404 - **Do not mix `drbd-utils` 9.x with the mainline kernel's 8.4 module**: the configuration405 syntax does not match.406- **DRBD is not a backup.** It replicates deletion and corruption at link speed.407408## 7. Operation, gates and prohibitions409410### 7.1 Operation411412- **Maintenance mode BEFORE touching anything.** `pcs property set maintenance-mode=true` (or413 `pcs node standby` / `crm node standby` for one node) before patching, restarting a service,414 testing something or looking with too much curiosity. **The most common cause of a cluster incident415 is an administrator operating the service by hand while the cluster watches.**416- **A cluster-managed resource is not touched with `systemctl`.** It is already forbidden in417 `linux-administration-standards`; it is confirmed here: `systemctl restart` on a managed418 resource makes the cluster see it as a failure and, depending on the configuration, a fencing.419- **Leaving maintenance is part of the procedure**, and you verify that the cluster sees the420 real state (`crm_mon -1`, no pending failures, a clean `pcs status`). A cluster that has spent421 weeks in `maintenance-mode` **is not providing HA** and nobody has noticed.422- **Rolling upgrades**, node by node, with the cluster running:423 - Pacemaker supports rolling upgrade **from 2.0.0 onwards**; from versions earlier than424 2.0.0 it is **not supported** — you have to go through a 2.x release first.425 - A node with Pacemaker 3.0+ **does not connect to Pacemaker Remote nodes of 1.1.14 or earlier**,426 and Pacemaker 1 does not talk to Remote/bundles of 3.0+. Take inventory before starting.427 - Pacemaker 3.0 **validates the CIB strictly**: `validate-with` is mandatory, case-sensitive, and428 does not accept old schemas (`pacemaker-1.1`, `pacemaker-next`, etc.). **A CIB429 that was working may fail to load after the upgrade.** Test it with `crm_verify`430 against the new schema before touching the first node.431- **Monitoring of the cluster FROM OUTSIDE the cluster.** A cluster that watches itself432 reports perfectly right up until the moment it can no longer report. The "cluster without433 quorum" or "node fenced" alert has to come out of a system that is not in the cluster434 (`observability-standards`). What to watch as a minimum:435 - quorum present and the expected number of nodes;436 - **`stonith-enabled` is `true`** (alert if somebody disables it — it happens);437 - resources stopped, failed or on an unexpected node;438 - **fencing failures** (a fencing that fails is a high-severity incident, even if the439 service stays up);440 - `maintenance-mode` active for more than X hours;441 - Corosync retransmissions and token loss (an early symptom of the network that will cause the442 next spurious fencing);443 - the state of the fencing devices: **the BMC/PDU is tested periodically, not when it is444 needed**;445 - the health of SBD and the watchdog.446- **The logs that matter**: `pacemaker.log` / the `pacemaker` and `corosync` journals, and above all447 **`crm_mon --show-detail` and the resource's failure history**. In an incident, the question448 is not "what happened" but **"why did the cluster decide this"**, and that answer is in the449 PE transition (`crm_simulate` over the transition file). Retention of those logs450 sufficient for a postmortem (`incident-management-standards`).451- **Fencing credentials**: they are "switch this server off" credentials. They go in the secrets452 manager (`secrets-management-standards`), with rotation, and their management network segmented.453454### 7.2 Exercises (recurring gates)4554561. **Before production**: the test of §4.3 — dirty power-off **and** network partition — with a457 report: what was switched off, how long the failover took, data integrity, what failed.4582. **A scheduled failover at least every six months**, in an agreed window, on the production459 cluster. A failover only tested on installation day is not tested: the kernel, the agents,460 the BMC firmware and the network rules have changed since then.4613. **Tested failback.** The way back is one more failover and is usually less rehearsed than the462 way out. It is exercised explicitly, and it is decided whether it is automatic (with low463 `stickiness`) or manual (recommended by default).4644. **A report per exercise**: date, scenario, measured failover time, deviation from465 the objective, findings and actions with an owner. Without a report, the exercise does not count.4665. **Testing the fencing devices separately** (`pcs stonith fence <node>` in a window):467 it confirms that the BMC answers, that the credentials are still valid and that the management468 network reaches it. It is the component that degrades most silently.4696. **`crm_simulate` in CI** over the versioned CIB on any constraint change.470471### 7.3 FORBIDDEN472473- ❌ **`stonith-enabled=false`** in anything other than a disposable lab474 documented as such. It is this document's number one prohibition.475- ❌ A cluster in production **without the dirty power-off test** of §4.3.476- ❌ **A two-node cluster with no qdevice/qnetd** (or without fencing that demonstrably resolves the477 partition).478- ❌ An arbiter (qnetd) in the same failure domain as a cluster node.479- ❌ **`no-quorum-policy=ignore`** copied without understanding what it enables.480- ❌ **A software watchdog (`softdog`) as fencing in production**: it is unreliable by design.481- ❌ Mounting a **non-cluster** filesystem (XFS, ext4, btrfs, ZFS) on two nodes at once.482- ❌ **Permanent DRBD dual-primary** under a filesystem that is not a cluster one.483- ❌ Confusing **replication with backup**, or **HA with DR** (§1.2).484- ❌ Operating a managed resource with `systemctl`, or touching the service without `maintenance-mode`.485- ❌ Editing `cib.xml` by hand on disk; configuring the cluster only through an interactive session without486 leaving the CIB versioned in the repository.487- ❌ Constraints with intermediate `score`s "to see what happens"; constraints without `crm_simulate`.488- ❌ A resource with no `monitor` operation, or with a `timeout` copied from the example.489- ❌ `resource-stickiness=0` (immediate automatic failback = a second free outage).490- ❌ **Exposing the `pcsd` daemon / its web interface to an untrusted network.** The bulk of the recent491 `pcs` CVEs comes from its **dependencies packaged in `pcsd`** (tornado, rack, lodash:492 RHSA-2026:2452 / 2462 / 2469 / 2818 / 2819, Feb 2026), not from the cluster logic. Management493 network, and patching at the pace of the distro's errata.494- ❌ Corosync over a network shared with backup or massive replication.495- ❌ **Pacemaker managing a host's containers** to "give them HA": see496 `podman-systemd-containers-standards`. The answer is an orchestrator, a load balancer with the497 state outside, or downtime accepted in writing.498- ❌ `keepalived`/VRRP in front of a stateful resource that cannot tolerate double writing.499- ❌ Building a cluster because "we want it not to go down", with no SLO, no failure domains500 drawn and nobody on call who knows how to operate it.501- ❌ Compiling Pacemaker/Corosync by hand on a system with vendor support.502- ❌ A new design on GFS2/OCFS2 without justifying its lifecycle in writing (§2).503504## 8. Mandatory web verification505506Before pinning anything in a real project:5075081. **The versions packaged by your distribution** of `pacemaker`, `corosync`, `pcs`/`crmsh`,509 `resource-agents`, `fence-agents` and `sbd` — **not upstream's**. In clusters, vendor510 support rules.5112. **The cluster filesystem's lifecycle** in your distro: GFS2 **discontinued from512 RHEL 10 onwards** (Resilient Storage Add-On; the `gfs2` and `dlm` modules out of the kernel), OCFS2513 deprecated in SLE HA 15 SP7 and **out of SLE HA 16**. Confirm the exact end-of-support dates514 for RHEL 9 before committing to a migration.5153. **Pacemaker 3.0 migration notes**: strict CIB validation, `validate-with`,516 withdrawn schemas, Pacemaker Remote compatibility. Read ClusterLabs's 3.0 changes page517 **before** the first rolling upgrade.5184. **The status of the specific fencing agent for your hardware/hypervisor/cloud** and its parameters519 (`pcmk_delay_base`, `pcmk_host_map`, `pcmk_reboot_action`): they change between versions of520 `fence-agents`.5215. **Security errata for `pcs`/`pcsd`** in your stream (the recent ones come from packaged522 dependencies) and CVEs for `corosync`, `pacemaker` and `sbd`.5236. **Patroni**: latest version and its compatibility matrix with the PostgreSQL version and with524 the chosen DCS; and whether it is still the default recommendation over Pacemaker.5257. **DRBD**: whether the 9.3 module has landed in the mainline kernel yet (LINBIT's estimate:526 possibly Linux 7.2, Sep/Oct 2026) — it completely changes the maintenance equation527 compared with DKMS.5288. **Corosync latency requirements and tuning** (`token`, `consensus`) for your topology, in529 particular if there are links between rooms.530531**Declared gaps — do NOT fill from memory, verify before using:**532- **Date and content of the last `sbd` release**: the last verified tag is **1.5.2533 (2023-01-09)** with repository activity in January 2026, but **it has not been confirmed whether534 a later release exists nor which version RHEL 10 and SLE HA 16 package**. Check it before535 depending on a specific fix.536- **The exact release date of `keepalived` 2.4.3** (tag verified, date not obtained because of an537 API rate limit) and its maintenance status.538- **The date of `drbd-utils` 9.34.0** (tag verified, date not obtained). And **the exact relationship539 between DRBD 9.3.3 and the 10.0.0 alpha branch**: LINBIT's support plan has not been verified.540- **The reason for the Patroni patch batch of 2026-07-07** (4.1.4, 4.0.10 and 3.3.11 on the same day,541 the typical pattern of a security fix): **not verified**. Consult the advisory before542 pinning a minimum version.543- **The status and support of `pg_auto_failover`** as an alternative to Patroni in two-node544 scenarios: mentioned in the sources but **not verified as to current maintenance**.545- **Supported alternatives to GFS2 on RHEL 10** for active/active over shared block:546 **not verified** beyond the observation that the add-on disappears. Confirm with Red547 Hat before designing.548- **The Corosync tuning parameters recommended today** (`token`, `token_retransmits_before_loss_const`,549 `consensus`): **not verified in this revision**. Do not copy values from blogs.550- **The exact compatibility of `crmsh` 5.1 with Pacemaker 3.0.x** (5.1.0 was at rc2 as of Jun 2026):551 not verified.552553If the web contradicts this document, **the web wins** — flag the discrepancy.