Site-to-Site VPN Diagram: IPsec Tunnels, BGP Routing and Redundancy

Your data center reaches its AWS workloads over a site-to-site VPN that has run quietly for a year. One night AWS replaces a tunnel endpoint during routine maintenance, and every connection to the VPC drops. Only tunnel 1 was ever configured on the firewall. Tunnel 2 had shown DOWN since the day it was created.

On most diagrams a site-to-site VPN is one line. In practice it is two IPsec tunnels, two BGP sessions and an MTU budget, and outages come from the parts the line hides. This post covers the IPsec building blocks, an AWS example, static routing versus BGP, redundancy, MTU and MSS, monitoring, and when a dedicated link fits better.

Site-to-site versus remote-access VPN

A site-to-site VPN joins two networks: gateways at each end hold the tunnels, and the hosts behind them need no VPN software. A remote-access VPN joins one device, with a client on each laptop and a user who signs in. If the real need is people reaching internal apps, an identity-aware proxy in a zero trust architecture often fits better than either.

The IPsec building blocks

IKEv2 (RFC 7296) authenticates the peers and negotiates keys. ESP (RFC 4303) encrypts the traffic.

IKEv2 normally sets up the IKE SA and the first child SA in four messages (IKE_SA_INIT and IKE_AUTH). IKEv1 was deprecated in 2023 by RFC 9395. For algorithms, RFC 8247 makes DH group 14 mandatory to implement and recommends group 19; groups 2, 5, 23 and 24 should not be used, and group 22 must not.

Two tunnels per connection: an AWS example

An AWS Site-to-Site VPN connects your customer gateway device to a gateway on the AWS side. Per the AWS tunnel options documentation, every connection has two tunnels, each with its own public IP, terminating in different Availability Zones.

AWS replaces tunnel endpoints for patches and hardware retirement, one tunnel at a time, so a device that uses only one tunnel can lose connectivity whenever that endpoint is replaced. AWS Health sends a monthly notice when one tunnel is down for more than an hour a day.

Each tunnel has an inside /30 from 169.254.0.0/16. AWS takes the first usable address, your device the second, and BGP runs between them. Your device starts IKE unless you set the startup action to start (IKEv2 only). AWS evaluates proposals from the lowest value in its allowed list, whatever order your device sends, so remove IKEv1, SHA-1 and DH groups 2, 5 and 22 to 24 on both sides.

Define the tunnels in code so both stay identical. In Terraform, tunnel 1:

resource "aws_vpn_connection" "dc" {
  vpn_gateway_id      = aws_vpn_gateway.main.id
  customer_gateway_id = aws_customer_gateway.dc.id
  type                = "ipsec.1"
  static_routes_only  = false

  tunnel1_inside_cidr                  = "169.254.10.0/30"
  tunnel1_ike_versions                 = ["ikev2"]
  tunnel1_phase1_encryption_algorithms = ["AES256-GCM-16"]
  tunnel1_phase1_integrity_algorithms  = ["SHA2-384"]
  tunnel1_phase1_dh_group_numbers      = [20]
  tunnel1_phase2_encryption_algorithms = ["AES256-GCM-16"]
  tunnel1_phase2_dh_group_numbers      = [20]
  tunnel1_dpd_timeout_seconds          = 30
  tunnel1_dpd_timeout_action           = "restart"
  tunnel1_startup_action               = "start"
  # tunnel2_*: the same values with 169.254.10.4/30
}

Static routing or BGP

Static routing lists your prefixes on the connection and leaves failover to your device's health checks. With BGP, each tunnel runs an eBGP session, and a dead tunnel's routes are withdrawn when its session drops. AWS recommends BGP wherever the device supports it, and ECMP requires it. Four rules:

1. Use distinct private ASNs. The AWS side defaults to 64512. Give your site its own private ASN: 64512 to 65534, or 4200000000 to 4294967294 (RFC 6996).
2. Advertise summaries. A virtual private gateway accepts at most 100 routes from your device, a fixed limit.
3. Never overlap CIDRs. The VPC's local route beats any VPN route, even a more specific one. Plan address space across sites, or NAT one side.
4. Know the tie-breakers. For the same prefix, a virtual private gateway prefers Direct Connect, then static VPN routes, then BGP routes; then the shortest AS path, then the lowest MED.

Without ECMP, AWS returns traffic over one tunnel at a time, marked with MED. The AWS route priority guidance is to allow asymmetric routing and keep AS paths equal. If your stateful firewall cannot cope, prepend the AS path on tunnel 2 and lower its local preference; a change of AWS egress path can then drop flows. The AWS example configuration uses a 10-second keepalive and 30-second hold time, not the 90 seconds RFC 4271 suggests.

Redundancy designs

Two tunnels cover failures on the AWS side. They still share your device, your ISP and your site.

With two devices, each has its own VPN connection and both advertise the same prefixes. ECMP hashes each flow onto one tunnel, so four tunnels give about 5 Gbps in total, but one TCP connection still tops out at 1.25 Gbps, and a stateful firewall may need source NAT for symmetry. With many sites, the transit gateway becomes the hub of a hub-and-spoke network.

MTU and MSS clamping

Each tunneled packet carries a new IP header plus ESP overhead. AWS supports at most a 1,446-byte MTU and a 1,406-byte TCP MSS, with no Path MTU Discovery and no jumbo frames. The exact values depend on your algorithms, per the AWS customer gateway best practices:

Set the tunnel MTU and clamp the MSS of TCP SYN packets in both directions, since each side announces its own. On a Linux gateway:

ip link set dev vti1 mtu 1446
iptables -t mangle -A FORWARD -o vti1 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1406
iptables -t mangle -A FORWARD -i vti1 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1406

The classic symptom of a wrong value: ping and SSH work, but file copies hang. The AWS device requirements page suggests 1,399 bytes, which also works but wastes a few bytes.

Monitoring the tunnels

• Tunnel state. CloudWatch publishes TunnelState per tunnel; for BGP connections, 1 means the session is established. Alarm when either tunnel drops below 1. TunnelDataIn and TunnelDataOut show which tunnel carries traffic.
• Logs. Site-to-Site VPN logs record IKE states, DPD events, BGP status and route changes.
• Your side. Watch BGP state and received prefixes per tunnel. An established session with zero prefixes is still an outage.

When a tunnel will not come up, check in layers: IKE SA, IPsec SA, a ping to the AWS inside address, then BGP.

When to move to Direct Connect

Move when traffic is steady in the gigabits, latency matters, or internet egress costs more than a port. Since September 2026, 10 and 100 Gbps dedicated connections can use flat-rate pricing that includes data transfer out within a tier. Keep the VPN as a backup: a virtual private gateway prefers Direct Connect routes, so traffic uses the tunnels only when the circuit fails. The disaster recovery architecture diagram covers that planning.

The site-to-site VPN diagram

How to read it:

• Left, the data center: 10.20.0.0/16 in AS 65010. The LAN sends VPC traffic to the edge firewall, the customer gateway device.
• Two tunnels reach two endpoints of the virtual private gateway, each with its own inside /30 and BGP session. Both are up; AWS picks the return tunnel.
• Right, the VPC: 10.50.0.0/16, with private subnets in two Availability Zones and a propagated route to 10.20.0.0/16.
• The Markdown block lists the settings that must match on both tunnels.
• The bottom notes say what two tunnels cover (endpoint replacements) and what they do not (one firewall, one ISP).

Drawing site-to-site VPN diagrams clearly

• Label both ends with CIDR, ASN and public IP. Most VPN tickets start with one of them.
• Draw both tunnels with their inside /30s and BGP peers, and say which one is preferred and why.
• Put the shared settings in one box: IKE version, algorithms, DH group, lifetimes, MTU and MSS.
• Show single points of failure, such as one device or one ISP.
• Leave out pre-shared keys, ISP internals and host lists.

The network topology diagram covers the office side; the AWS VPC diagram covers subnets and route tables.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Frame from the Core pack for the data center, named with its CIDR and ASN. Inside it, add a Subnet with two Server nodes and a Firewall, all from the Network pack.
3. Add a VPC from the Network pack. Inside it, add a Frame for the virtual private gateway with two Router nodes, and two private Subnet nodes with a Compute node from the Cloud pack in each.
4. Draw the tunnels from the firewall to the endpoints with arrows at both ends, labeled with the inside /30 and BGP peers. Add edges from the LAN to the firewall and from the gateway to the subnets.
5. Add a Markdown block from the Core pack between the tunnels for the shared settings, and Text notes.

If the VPN is defined in Terraform, the free Terraform to diagram tool gives you a first draft; see also network diagrams.

The example uses 14 diagram nodes, within the Free plan.

FAQ

What should a site-to-site VPN diagram show? Both networks with their CIDRs, each gateway's public IP and ASN, every tunnel's inside addresses, the routing, and the settings both sides must match.

Why does an AWS Site-to-Site VPN have two tunnels? Each connection has two tunnels to endpoints in different Availability Zones. AWS maintains them one at a time, so with both configured, traffic moves instead of stopping.

Should I use static routing or BGP over IPsec? BGP, whenever your device supports it. It withdraws a failed tunnel's routes automatically and is required for ECMP. Static routing is for devices without BGP.

What MTU and MSS should I use for an AWS VPN? At most 1,446 and 1,406 bytes, with AES-GCM and no NAT traversal. CBC ciphers and NAT-T need smaller values from the AWS table.

When should I use Direct Connect instead of a VPN? When traffic is steady in the gigabits, latency must be predictable, or you need up to a 99.99% SLA. Keep the VPN as an encrypted backup.

Start free — one seat, no credit card.