vahagn-madatyan IPSec/IKE VPN Troubleshooting
State-machine-driven troubleshooting skill for IPSec VPN tunnels using IKE negotiation diagnosis. IKE is a protocol state machine — IKEv1 Main Mode progresses through defined states (MM_NO_STATE → MM_SA_SETUP → MM_KEY_EXCH → MM_KEY_AUTH → QM_IDLE), and IKEv2 uses SA_INIT/SA_AUTH exchanges. Reading the current SA state and mapping it to the negotiation phase isolates the failure domain without guesswork.
Commands are labeled [Cisco], [JunOS], [PAN-OS], or [FortiGate]
where syntax diverges. Unlabeled statements apply to all four vendors. Reference
references/state-machine.md for full IKE FSM detail and
references/cli-reference.md for complete command tables.
When to Use
- VPN tunnel reported down or not passing traffic
- IKE SA stuck in a non-established state (MM_NO_STATE, AG_INIT_EXCH, SA_INIT_SENT)
- Phase 1 establishes but Phase 2 / Child SA fails to negotiate
- Tunnel is established but traffic is not flowing (encap/decap counters not incrementing)
- After configuration changes to crypto proposals, peer addresses, or PSK/certificates
- Intermittent tunnel flapping or unexpected SA rekey failures
- NAT-related VPN issues (NAT-T not activating, keepalive failures)
- Dead Peer Detection declaring a peer dead prematurely
- Crypto mismatch diagnosis after multi-vendor interop changes
Prerequisites
- SSH or console access to the VPN gateway (read-only privilege sufficient)
- At least one IPSec VPN tunnel configured on the device
- Knowledge of expected tunnel topology: peer IP addresses, protected networks (proxy IDs / traffic selectors), and intended crypto parameters
- Awareness of configured IKE version (IKEv1 vs IKEv2) per tunnel
- For route-based VPN: knowledge of tunnel interface assignments and routing
- For policy-based VPN: knowledge of crypto map or security policy bindings
Procedure
Follow this diagnostic flow sequentially. Each step builds on data from prior steps. The procedure moves from SA state inventory through targeted failure diagnosis to tunnel health verification.
Step 1: Check IKE SA State
Collect all IKE SAs and compare against expected tunnel topology.
[Cisco]
show crypto isakmp sa
show crypto ikev2 sa
[JunOS]
show security ike security-associations
[PAN-OS]
show vpn ike-sa
[FortiGate]
diagnose vpn ike gateway list
Record each SA: peer address, IKE version, state, role (initiator/responder), lifetime remaining. Compare against expected topology — every configured tunnel peer should have an IKE SA. Missing SAs mean the tunnel was never initiated or was cleared. Any SA not showing an established state requires Step 2 diagnosis.
For IKEv1, map the Cisco state name to the negotiation phase:
- MM_NO_STATE → no exchange started or failed at SA proposal
- MM_SA_SETUP → SA proposal accepted, DH exchange next
- MM_KEY_EXCH → DH complete, authentication next
- MM_KEY_AUTH → authenticated, completing
- QM_IDLE → Phase 1 complete, ready for Phase 2
Step 2: Diagnose Stuck or Failed State
For any SA not in established state, the IKE state reveals the failure domain.
IKEv1 Main Mode failure isolation:
- Stuck at MM_NO_STATE: No response from peer. Check: peer reachable on UDP 500, firewall permits IKE, peer has matching crypto map, peer IKE process running.
- Stuck at MM_SA_SETUP: Proposal mismatch. The peers agreed on an SA but the DH key exchange failed. Check: DH group mismatch between proposals.
- Stuck at MM_KEY_EXCH: Authentication failure. DH succeeded but identity verification failed. Check: PSK mismatch, certificate chain invalid, peer ID type mismatch (IP vs FQDN).
- QM_IDLE with no Phase 2: Phase 1 established but no IPSec SA created. Check: no interesting traffic triggering the tunnel, proxy ID mismatch, transform set mismatch.
IKEv2 failure isolation:
- No SA_INIT response: Peer unreachable on UDP 500/4500 or no matching IKE proposal.
- NO_PROPOSAL_CHOSEN notify: No overlapping crypto suite. Compare encryption, PRF, integrity, and DH group on both sides.
- INVALID_KE_PAYLOAD notify: DH group mismatch. The responder returns its preferred group; initiator should retry automatically.
- AUTHENTICATION_FAILED notify: PSK mismatch or certificate validation failure.
- TS_UNACCEPTABLE notify: Traffic selector (proxy ID) mismatch between peers.
Check SA detail for error messages and last failure reason:
[Cisco]
show crypto isakmp sa detail
show crypto ikev2 sa detail
[JunOS]
show security ike security-associations detail
[PAN-OS]
show vpn ike-sa gateway <name>
[FortiGate]
diagnose vpn ike gateway list name <name>
Step 3: Verify Crypto Parameter Alignment
For proposal mismatches, compare configured crypto parameters across both peers.
Phase 1 / IKE parameters to verify:
| Parameter | Must Match? | Notes |
|---|---|---|
| Encryption algorithm | Yes | AES-128, AES-256, 3DES — must be identical |
| Hash / integrity | Yes | SHA-256, SHA-384, SHA-512 — must be identical |
| DH group | Yes | Group 14 (2048-bit), 19 (256-bit ECP), 20 (384-bit ECP) |
| Authentication method | Yes | PSK, RSA-sig, ECDSA — must agree |
| SA lifetime | Negotiable | Shorter value wins; large discrepancies cause rekey issues |
| IKE version | Yes | Both peers must use the same version |
[Cisco]
show crypto isakmp policy
show crypto ikev2 proposal
[JunOS]
show security ike proposal
[PAN-OS]
show vpn ike crypto-profiles
[FortiGate]
diagnose vpn ike config list
Verify Phase 2 / IPSec parameters similarly: encryption, hash, PFS DH group, and SA lifetime must align. Proxy IDs (traffic selectors) must match or overlap.
Step 4: Validate Phase 2 / Child SA
With Phase 1 established, verify IPSec SAs are created and functional.
[Cisco]
show crypto ipsec sa
[JunOS]
show security ipsec security-associations
[PAN-OS]
show vpn ipsec-sa
[FortiGate]
diagnose vpn tunnel list
Verify for each tunnel: SA state (active), SPI values (inbound and outbound present), proxy IDs / traffic selectors match intended protected networks, encapsulation mode (tunnel or transport), and PFS status.
Common Phase 2 failures:
- Transform set mismatch: encryption or hash algorithm differs
- Proxy ID mismatch: local/remote network definitions do not match between peers — the most common interop failure
- PFS DH group mismatch: one side requires PFS, other has it disabled or uses a different DH group
- No interesting traffic: policy-based VPN requires matching traffic to trigger Quick Mode
Step 5: Assess Tunnel Health
For established tunnels, evaluate operational health indicators.
Counter analysis: Check encap/decap counters in IPSec SA output. Both should increment steadily.
| Condition | Interpretation |
|---|---|
| Encap incrementing, decap zero | Outbound traffic sent but nothing returned — remote peer issue, routing, or ACL |
| Decap incrementing, encap zero | Receiving traffic but not sending — local routing or crypto ACL issue |
| Both incrementing | Tunnel passing traffic in both directions |
| Both zero | Tunnel established but no traffic matched — verify routing and proxy IDs |
| Encrypt/decrypt errors rising | Crypto processing failures — check for hardware crypto engine issues |
| Replay failures | Anti-replay window exceeded — packet reordering, QoS, or async routing |
DPD health: Verify DPD packets are exchanged. If DPD declares a peer dead, check the underlying transport path — DPD failures indicate loss of reachability, not a crypto problem.
Rekey status: Verify SAs are rekeying before expiry. A tunnel that fails to rekey will drop traffic when the SA expires. Check lifetime remaining and compare against rekey margin.
NAT-T health: If NAT-T is active (UDP 4500), verify NAT keepalive interval is shorter than the NAT device's UDP session timeout. Typical NAT timeout is 60–300s; keepalive should be 10–30s.
Step 6: Generate Report
Compile findings using the Report Template below. Classify each finding by severity and include the IKE state or counter evidence that supports the diagnosis.
Threshold Tables
Operational parameter norms for IPSec/IKE VPN — protocol-level expectations.
| Parameter | Typical Default | Warning | Critical |
|---|---|---|---|
| IKE SA Lifetime | 86400s (24h) | <10% remaining without rekey initiated | Expired without rekey |
| IPSec SA Lifetime | 3600s (1h) | <10% remaining without rekey initiated | Expired without rekey |
| DPD Interval | 10–30s | >60s (slow detection) | DPD disabled on critical tunnels |
| DPD Retries | 3–5 | >10 (slow failover) | 0 (DPD disabled) |
| NAT-T Keepalive | 20s | >NAT timeout / 2 | >NAT session timeout |
| Rekey Margin | 540s (IKE), 270s (IPSec) | <60s (tight rekey window) | 0s (rekey at expiry) |
Counter Thresholds:
| Counter | Healthy | Warning | Critical |
|---|---|---|---|
| Encap packets | Incrementing | Flat for >5 min | Zero after tunnel up >10 min |
| Decap packets | Incrementing | Flat for >5 min | Zero after tunnel up >10 min |
| Encrypt errors | 0 | >0 but <0.1% of encap | >1% of encap |
| Decrypt errors | 0 | >0 but <0.1% of decap | >1% of decap |
| Replay failures | 0 | >0 occasional | Sustained >10/min |
| Rekey failures | 0 | 1 (recovered) | >1 consecutive |
Crypto Strength Minimums (current best practice):
| Parameter | Minimum | Recommended | Avoid |
|---|---|---|---|
| Encryption | AES-128 | AES-256 | DES, 3DES |
| Hash / PRF | SHA-256 | SHA-384+ | MD5, SHA-1 |
| DH Group | 14 (2048-bit) | 19/20 (ECP 256/384) | 1, 2, 5 |
Decision Trees
IKE SA State Triage
VPN tunnel not working
├── No IKE SA exists
│ ├── Tunnel never configured? → Verify crypto map / IKE gateway config
│ ├── Peer unreachable? → Ping peer, check UDP 500 path
│ ├── No interesting traffic? → (Policy-based) verify crypto ACL matches
│ └── SA cleared / expired? → Check logs for delete or expiry events
│
├── IKE SA exists but NOT established
│ ├── IKEv1 MM_NO_STATE / AG_NO_STATE
│ │ ├── No response from peer → UDP 500 blocked or peer not configured
│ │ └── Proposal mismatch → Compare ISAKMP policies on both sides
│ │
│ ├── IKEv1 MM_SA_SETUP (stuck)
│ │ └── DH group mismatch → Align DH groups in proposals
│ │
│ ├── IKEv1 MM_KEY_EXCH (stuck)
│ │ ├── PSK mismatch → Verify pre-shared keys on both peers
│ │ ├── Certificate failure → Check cert chain, expiry, CRL
│ │ └── ID type mismatch → Verify peer ID (IP vs FQDN)
│ │
│ ├── IKEv2 NO_PROPOSAL_CHOSEN
│ │ └── No overlapping crypto suite → Align enc, PRF, integrity, DH
│ │
│ ├── IKEv2 AUTHENTICATION_FAILED
│ │ ├── PSK mismatch → Verify keys
│ │ └── Certificate error → Check chain, OCSP/CRL, SAN
│ │
│ └── IKEv2 TS_UNACCEPTABLE
│ └── Traffic selector mismatch → Align proxy IDs / encryption domains
│
├── IKE SA established, no IPSec SA (Phase 2 failure)
│ ├── Transform set mismatch → Compare Phase 2 proposals
│ ├── Proxy ID mismatch → Compare local/remote network definitions
│ ├── PFS DH group mismatch → Align or disable PFS on both sides
│ └── No traffic trigger → (Policy-based) send matching traffic
│
└── Both SAs established, no traffic
├── Routing? → Verify routes point through tunnel interface
├── Encap up, decap zero → Remote peer or return-path issue
├── Both counters zero → No matching traffic; check ACL/policy
└── Errors incrementing → Check crypto engine, replay, MTU
Tunnel Flapping Triage
Tunnel repeatedly drops and re-establishes
├── DPD declaring peer dead
│ ├── Underlying transport unstable → Check physical link, ISP circuit
│ ├── QoS dropping IKE keepalives → Verify DSCP marking for IKE traffic
│ └── DPD timer too aggressive → Increase interval or retries
│
├── Rekey failure
│ ├── Lifetime mismatch → Both sides rekey at different times
│ ├── Simultaneous rekey collision → One side should be initiator-only
│ └── Crypto engine overloaded → Check CPU during rekey burst
│
├── Certificate expiry
│ └── Cert expired mid-session → Monitor cert validity, automate renewal
│
└── NAT mapping expiry
├── Keepalive > NAT timeout → Reduce NAT-T keepalive interval
└── NAT device rebooted → SA stale until DPD detects and renegotiates
Report Template
VPN/IPSEC TROUBLESHOOTING REPORT
==================================
Device: [hostname]
Vendor: [Cisco | JunOS | PAN-OS | FortiGate]
IKE Version: [IKEv1 | IKEv2]
Check Time: [timestamp]
Performed By: [operator/agent]
TUNNEL INVENTORY:
- Total configured tunnels: [n]
- IKE SAs established: [n] | Failed/Missing: [n]
- IPSec SAs active: [n] | Failed/Missing: [n]
- Tunnels requiring attention: [list with states]
FINDINGS:
1. [Severity] [Category] — [Description]
Tunnel: [peer address / tunnel name]
IKE State: [current state]
Observed: [state, counter, or error]
Expected: [normal state or value]
Root Cause: [diagnosis from decision tree]
Action: [recommended remediation]
CRYPTO PARAMETER SUMMARY:
- IKE: [encryption] / [hash] / [DH group] / [auth method] / [lifetime]
- IPSec: [encryption] / [hash] / [PFS group] / [lifetime]
- Strength Assessment: [meets minimum | below minimum — specify gap]
TUNNEL HEALTH:
- Encap/Decap: [incrementing | flat | zero]
- Errors: [count and type]
- DPD: [active | disabled | peer-dead events]
- NAT-T: [active on port 4500 | not detected | misconfigured]
- Last Rekey: [timestamp] | Next Rekey: [timestamp]
RECOMMENDATIONS:
- [Prioritized action list]
NEXT CHECK: [CRITICAL: 1hr, WARNING: 8hr, HEALTHY: 24hr]
Troubleshooting
Proxy ID / Traffic Selector Mismatch
The most common multi-vendor interop failure. Each vendor expresses protected networks differently. Cisco uses crypto ACLs, JunOS uses address-book entries, PAN-OS uses proxy IDs in the tunnel config, FortiGate uses phase2 selectors. Compare the effective local/remote network pairs on both sides — they must mirror (local on on the other). For IKEv2, check TS_UNACCEPTABLE notify in logs.
NAT-T Not Activating
If NAT exists between peers but tunnels use ESP (protocol 50) instead of UDP
4500, verify NAT-T is enabled on both sides. [Cisco] requires
crypto isakmp nat-traversal; [FortiGate] enables it by default;
[JunOS] and [PAN-OS] enable it automatically when NAT is detected.
Firewalls between peers must allow both UDP 500 and UDP 4500.
DPD Premature Failover
DPD declares peer dead when keepalive probes are lost. If the tunnel is
functional but DPD triggers, the underlying path may have transient loss or
QoS may deprioritize IKE/ESP. Increase the DPD retry count or interval.
Verify the DPD mode: [Cisco] on-demand only sends probes when there is
outbound traffic; periodic sends probes regardless.
Rekey Failures
If a tunnel drops at SA expiry instead of rekeying, check: (1) lifetime mismatch causing both sides to rekey simultaneously, (2) IKE SA expired before IPSec SA rekey completed (IKE SA must be active to negotiate new IPSec SA), (3) crypto engine busy (high tunnel count causing rekey storm). Stagger lifetimes by 5–10% between peers to avoid simultaneous rekey.
MTU / Fragmentation Issues
IPSec overhead (ESP header, IV, padding, ICV) reduces effective MTU. With
AES-256/SHA-256 in tunnel mode, overhead is approximately 73 bytes. If
traffic requires near-MTU packets, set tunnel interface MTU to 1400 and
TCP MSS to 1360. Symptoms of MTU issues: small packets pass, large transfers
stall or fail. [Cisco] crypto ipsec df-bit clear clears the DF bit to
allow fragmentation. Pre-fragmentation (before encryption) is preferred over
post-fragmentation.