Computer Networking › Section H · Security in computer networks › Topic H.7

IPsec and virtual private networks

TLS protects one application's connection at a time. IPsec works one layer lower and protects every datagram travelling between two hosts or gateways, which lets an organisation build a private network out of public internet links.

Security for every datagram

IPsec is a suite of protocols that secures traffic at the network layer. Once two IPsec devices have set up protection, every datagram between them is covered: TCP segments, UDP datagrams and ICMP messages alike, whatever application produced them. Applications need no changes and need not even know IPsec is present. IPsec can provide confidentiality, data integrity, authentication of the sending device and protection against replayed datagrams. The two ends can be two hosts, a host and a gateway, or two gateways, and the gateway case is the basis of most virtual private networks.

Virtual private networks

A company with offices in several cities once had two choices: lease private lines between them, which is costly, or send internal traffic across the public internet, where anyone on the path could read it. A virtual private network (VPN) gives the privacy of the first at the cost of the second. Each office has a gateway router. When a datagram leaves one office for another, the gateway encrypts and authenticates it and sends it across the internet to the other office's gateway, which checks and decrypts it. Remote-access VPNs do the same between a travelling employee's laptop and the company's gateway.

AH and ESP; transport and tunnel mode

IPsec offers two protocols. The Authentication Header (AH) provides source authentication and integrity but no confidentiality. The Encapsulating Security Payload (ESP) provides confidentiality as well as integrity and authentication, so it is used far more widely; AH also clashes with NAT, because it protects the address fields that NAT rewrites. Each can run in two modes. Transport mode protects the payload of a datagram between two hosts and keeps the original IP header. Tunnel mode protects the whole original datagram, header included, and wraps it in a new datagram between gateways. VPNs normally use ESP in tunnel mode.

Security associations

Before sending protected traffic, the two ends set up a security association (SA), a one-way logical connection. Two-way traffic needs two SAs, one for each direction. Each SA is identified by a 32-bit Security Parameter Index (SPI) and records the endpoints, the encryption and integrity algorithms, their keys, a sequence number counter and a lifetime. A device stores its SAs in its Security Association Database. A separate Security Policy Database says what to do with each kind of traffic: protect it with a given SA, let it bypass IPsec, or discard it. For example, traffic between the two offices' subnets is protected, and everything else is sent normally.

Inside an ESP tunnel-mode datagram

Reading from the front, the protected datagram contains: 1. A new IP header with the two gateways' addresses and protocol number 50, meaning ESP. 2. The ESP header: the SPI and a 32-bit sequence number. 3. For many ciphers, an initialisation vector. 4. The encrypted part: the entire original datagram followed by the ESP trailer, which holds padding, the pad length and a next-header field naming what is inside, an IPv4 datagram. 5. The ESP integrity check value (ICV), a MAC over the ESP header and encrypted part. Routers on the path see only the outer header, so even the inner addresses are hidden.

Worked example: a datagram between two offices

Host 10.1.0.5 in the branch office sends a datagram to 10.2.0.9 at head office. The gateways are 198.51.100.1 and 203.0.113.1, documentation addresses standing in for public ones. 1. The branch gateway checks its policy database: traffic from 10.1.0.0/16 to 10.2.0.0/16 must use its SA towards head office. 2. It adds the trailer, encrypts the datagram, adds the ESP header with the SA's SPI and next sequence number, computes the ICV and adds an outer header from 198.51.100.1 to 203.0.113.1. 3. Head office's gateway uses the SPI to find the SA, checks the sequence number and ICV, decrypts, and forwards the original datagram. Check: the internet saw only gateway-to-gateway ESP.

Worked example: the cost in bytes

Assume ESP in tunnel mode with AES-GCM: a 20-byte outer IPv4 header, an 8-byte ESP header, an 8-byte IV and a 16-byte ICV. The original datagram plus 2 trailer bytes must be padded to a multiple of 4 bytes. 1. Take a 1,400-byte original datagram: 1,400 + 2 = 1,402, which needs 2 bytes of padding to reach 1,404. 2. Total: 20 + 8 + 8 + 1,404 + 16 = 1,456 bytes, which fits a 1,500-byte Ethernet MTU. 3. A 1,480-byte datagram gives 20 + 8 + 8 + 1,484 + 16 = 1,536 bytes, too big. Check: the overhead is 56 bytes in both cases, which is why tunnels lower the usable MTU.

IKE: setting up SAs automatically

Typing keys and SAs into every device by hand is impractical for more than a few sites and leaves keys unchanged for too long. The Internet Key Exchange protocol, now IKEv2, automates it. It runs over UDP port 500, or port 4500 when a NAT is on the path. First the two ends create an IKE SA: they exchange Diffie-Hellman values to derive keys, then authenticate each other with certificates or a pre-shared key. Protected by that, they negotiate child SAs, the ESP SAs that carry data. IKE also replaces keys regularly, so a single key never protects too much traffic.

VPNs today

IPsec remains common for site-to-site links, but it has company. WireGuard, added to the Linux kernel in 2020, is a much smaller VPN protocol with a fixed set of modern algorithms, including Curve25519 and ChaCha20-Poly1305, which makes it easier to audit and configure. TLS-based VPNs tunnel traffic inside TLS connections. Many organisations are also moving towards zero trust, which stops treating anything inside the VPN as automatically trustworthy and checks every request. Consumer VPN services encrypt traffic as far as the provider's server, so they move trust to that provider rather than removing the need for it.

Common mistakes to avoid

First, assuming AH encrypts data; only ESP provides confidentiality. Second, forgetting that an SA is one-way, so two-way traffic needs two. Third, mixing up the modes: tunnel mode wraps the whole original datagram in a new one, while transport mode keeps the original header. Fourth, thinking routers on the path can read the inner addresses of a tunnel. Fifth, ignoring tunnel overhead, which can push full-size datagrams past the MTU. Sixth, assuming a VPN makes the far end trustworthy; it protects only the path between the tunnel ends.

Key takeaways

IPsec secures every datagram between two hosts or gateways, without changing applications. A VPN uses IPsec tunnels across the public internet to join offices or remote users to a private network. ESP provides confidentiality, integrity and authentication; AH lacks confidentiality. Tunnel mode wraps the whole original datagram in a new one with a new IP header, ESP header, encrypted datagram and trailer, and an ICV. One-way security associations, named by SPIs, hold algorithms, keys and sequence numbers. IKEv2 sets up and refreshes SAs using Diffie-Hellman and mutual authentication. WireGuard and TLS-based VPNs are common alternatives.

Flashcards

Flashcard

IPsec

Network-layer security for every datagram between two devices

Applications need no changes.

Test your understanding

Question 1 of 5

A company needs its inter-office traffic to be unreadable on the internet. Which IPsec choice is standard for this?

Question 2 of 5

Two hosts exchange IPsec-protected traffic in both directions. How many security associations are needed at minimum?

Question 3 of 5

With ESP tunnel mode using a 20-byte outer header, 8-byte ESP header, 8-byte IV and 16-byte ICV, and padding so that datagram plus 2 trailer bytes is a multiple of 4, how big does a 1,000-byte datagram become?

Question 4 of 5

A router in the middle of the internet forwards an ESP tunnel-mode datagram between two gateways. What can it see?

Question 5 of 5

What does IKEv2 do in an IPsec deployment?

← Previous topicNext topic →