What Is an IPSec VPN?

IPSec operates at the network layer (Layer 3) of the OSI model, which means it encrypts IP packets directly — regardless of the application generating them. This is different from TLS/SSL (used by OpenVPN), which operates at the transport layer.

The IPSec process:

1. IKE Phase 1 (Authentication) — The two endpoints (your device and the VPN server) authenticate each other using pre-shared keys, digital certificates, or EAP (Extensible Authentication Protocol). They establish a secure channel called the IKE Security Association (SA)

1. IKE Phase 2 (Key Exchange) — Using the secure channel from Phase 1, both sides negotiate encryption parameters: which cipher to use (AES-256), which integrity algorithm (SHA-256), and which Diffie-Hellman group for key exchange. They generate session-specific encryption keys

1. Data Transfer — With the IPSec Security Association established, all traffic between the endpoints is encrypted and authenticated. Each packet is encapsulated, encrypted, and tagged with an integrity check

1. Rekeying — Periodically, the encryption keys are rotated (typically every 1–8 hours) to maintain perfect forward secrecy. If one key is compromised, only that time window's data is affected

The entire setup process takes 1–3 seconds with IKEv2. Older implementations using IKEv1 could take longer and were more prone to connection issues.

IPSec is actually a suite of three sub-protocols:

ESP (Encapsulating Security Payload) — The workhorse of IPSec. ESP provides: - Encryption — Encrypts the packet payload (data) using AES-256 or other ciphers - Authentication — Verifies packet integrity and origin - Anti-replay protection — Prevents attackers from capturing and resending packets

ESP uses protocol number 50 and is the sub-protocol used by virtually all modern IPSec VPN implementations.

AH (Authentication Header) — Provides authentication and integrity but NOT encryption. AH verifies that packets haven't been tampered with and confirms the sender's identity, but the data itself is sent in cleartext. AH uses protocol number 51 and is rarely used alone in practice — ESP covers its functionality plus encryption.

IKE (Internet Key Exchange) — Manages the negotiation and establishment of Security Associations. IKE handles authentication, key exchange, and parameter negotiation. IKEv2 (the current version) includes built-in NAT Traversal, MOBIKE for mobile network switching, and improved reliability over IKEv1.

In consumer VPN contexts, when you see "IKEv2/IPSec" as a protocol option, it means IKEv2 handles the connection setup and ESP handles the actual data encryption.

IPSec operates in two distinct modes:

Tunnel Mode — The entire original IP packet (header + data) is encrypted and placed inside a new IP packet with a new header. The outer header is addressed to the VPN endpoint.

• Used by: Site-to-site VPNs, consumer VPN connections
• The original source and destination IP addresses are hidden inside the encrypted payload
• The outside world only sees traffic between the two VPN endpoints
• This is the mode used by IKEv2/IPSec in consumer VPN apps

Transport Mode — Only the packet's data payload is encrypted. The original IP header is preserved and visible.

• Used by: Host-to-host encryption (e.g., between two servers on the same network)
• Source and destination IP addresses remain visible
• Lower overhead than tunnel mode (no second header)
• Typically used within trusted networks for server-to-server encryption

For VPN use, tunnel mode is almost always what's deployed. It provides complete packet encryption and hides the original routing information. Transport mode is used in specialized enterprise scenarios where the endpoints are on the same network segment.

How does IPSec compare to other VPN technologies?

IPSec/IKEv2 vs. WireGuard: - IPSec: Mature, built into all operating systems, complex but battle-tested - WireGuard: Newer, faster (~varies speed advantage), simpler codebase - Better fit: WireGuard for speed and simplicity; IKEv2/IPSec for mobile network switching (MOBIKE)

IPSec vs. OpenVPN: - IPSec: Operates at network layer, built-in OS support, faster - OpenVPN: Operates at transport layer, runs on any port (harder to block), more configurable - Better fit: OpenVPN for firewall evasion; IPSec for speed and native integration

IPSec vs. SSL/TLS VPN: - IPSec: Encrypts all IP traffic, requires client software or OS support - SSL/TLS: Works through web browsers, easier to deploy for web-based access - Better fit: IPSec for full network access; SSL/TLS for browser-based access to specific applications

For consumer VPN users, the choice between IKEv2/IPSec and WireGuard is the most relevant comparison. WireGuard is faster and more efficient, but IKEv2/IPSec handles mobile network transitions more gracefully. Many VPN providers offer both — use WireGuard as default and IKEv2 as a fallback.

IPSec remains one of the most widely deployed VPN technologies:

Enterprise Site-to-Site VPNs — The primary use case. Companies use IPSec tunnels to connect office locations, data centers, and cloud infrastructure. Cisco, Palo Alto, Fortinet, and Juniper all use IPSec as their primary site-to-site VPN protocol.

Cloud VPN Gateways — AWS VPN, Azure VPN Gateway, and Google Cloud VPN all use IPSec for connecting on-premises networks to cloud infrastructure.

Mobile VPN Connections — IKEv2/IPSec is built into iOS, Android, Windows, and macOS. Many consumer VPN providers offer it as a protocol option, particularly for mobile devices where its MOBIKE capability provides seamless network switching.

IoT and Embedded Devices — IPSec's inclusion in the IP stack makes it suitable for embedded devices and IoT applications where installing a full VPN client isn't feasible.

Government and Military — IPSec with AES-256 encryption meets FIPS 140-2/140-3 compliance requirements, making it the standard for government communications.

While WireGuard is gaining ground in the consumer VPN space, IPSec remains dominant in enterprise and infrastructure contexts due to its maturity, hardware acceleration support, and universal vendor compatibility.

IPSec is considered highly secure when properly configured, but there are important considerations:

Strengths: - AES-256 encryption — Considered unbreakable with current technology - Perfect forward secrecy — Session keys are ephemeral, protecting past sessions if a long-term key is compromised - Hardware acceleration — Most modern CPUs and network equipment include dedicated AES-NI instructions, enabling line-rate encryption - Mature implementation — Decades of testing, auditing, and real-world deployment

Potential weaknesses: - Implementation complexity — IPSec's complexity means more potential for configuration errors. A misconfigured IPSec setup can be significantly weaker than intended - Pre-shared keys — If a PSK is used (instead of certificates) and the PSK is weak, the entire VPN can be compromised. Always use strong, random PSKs or certificate-based authentication - Vendor-specific extensions — Some vendors add proprietary extensions to IPSec that may not be publicly documented

Best practices: - Use IKEv2 (not IKEv1 — the older version has known weaknesses) - Use AES-256-GCM for encryption (not 3DES or AES-CBC) - Use SHA-256 or SHA-384 for integrity (not MD5 or SHA-1) - Enable perfect forward secrecy with Diffie-Hellman Group 14 or higher - Use certificate-based authentication when possible