HAProxy as a self-healing internet gatekeeper

Overview

HAProxy sits as the single front door for all web traffic — terminating TLS, applying access lists, load-balancing to backend services, and logging every request. Two identical nodes run behind a keepalived daemon: one is master, one is standby, and if the master fails, keepalived promotes the standby and moves connectivity over with it, rather than leaving a dangling DNS record pointed at a dead host.

                 [ Internet ]              [ Internal network ]
                       │                          │
              External Elastic IP        Internal Elastic IP
                       │                          │
                       ▼                          ▼
              ┌──────────────────────────────────────┐
              │   HAProxy — MASTER (keepalived: active)│  ── DMZ
              └──────────────────────────────────────┘
                       │  keepalived health check
                       ▼
              ┌──────────────────────────────────────┐
              │   HAProxy — STANDBY (keepalived: idle) │  ── DMZ
              └──────────────────────────────────────┘
                       │
                       ▼
              [ Service ]  [ Service ]  [ Service ]   ── PROD

1. Why Elastic IP failover instead of a floating internal VIP

The usual keepalived pattern uses VRRP to move a virtual IP between two hosts on the same subnet. On AWS, the equivalent is reassigning an Elastic IP to whichever instance is currently active — done from both sides at once, so HAProxy is reachable as a single logical entity whether the request originates from the internet or from the internal network:

• External Elastic IP — the internet-facing address DNS actually points at
• Internal Elastic IP — a second address reachable from the internal network, so internal clients don't have to route out through the internet-facing path

Both addresses move together when keepalived decides the standby node should take over — from DNS's point of view, and from every client's point of view, nothing changed except which physical instance is now answering.

2. Node layout

IAM: least-privilege Elastic IP reassignment

keepalived triggers a script on failover that calls the AWS API to re-associate both Elastic IPs to the newly active instance. That call runs as a dedicated IAM user, scoped to only the ability to associate/disassociate Elastic IPs — not general EC2 administration:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

3. SELinux hardening

Both nodes run with SELinux enforcing, not permissive or disabled — HAProxy is internet-facing, so this isn't optional. One specific boolean needs to be enabled, or HAProxy loses the ability to reach backends immediately after an Elastic IP reassignment changes the active network interface's addressing:

# Without this, SELinux blocks the "name_connect" action once the
# Elastic IP changes on the network interface — HAProxy can bind but
# can't actually reach upstream services after a failover.
setsebool -P haproxy_connect_any on

4. Package version note

On older CentOS/RHEL-family systems, the official repository ships a dated HAProxy release; a newer one is typically available from EPEL, but under a versioned package/service name — worth checking explicitly rather than assuming haproxy is the current one:

# Reload after a config change
systemctl reload haproxy18

# Validate config syntax before reloading, without actually applying it
haproxy18 -d -V -c -f /etc/haproxy18/haproxy.cfg

5. Certificate lifecycle

TLS termination happens at HAProxy, so certificate renewal has to land identically on both nodes — a renewal that only reaches the master and never gets synced to the standby is a failure waiting for the next failover to expose it.

• Certificates are issued and renewed via Let's Encrypt / certbot
• The renewed bundle is stored at the same path on both nodes (e.g. /etc/haproxy18/ssl)
• A renewal hook reloads HAProxy on both nodes after a successful issuance, so neither node is ever serving a stale certificate

6. Zone layout

Both HAProxy nodes sit in a DMZ segment, separate from the production services they load-balance to — HAProxy terminates the internet-facing connection and speaks to backends over an internal, non-internet-routable path:

Internet ── HTTP/HTTPS ──►  DMZ  [ HAProxy master / standby ]
                                        │
                                        ▼
                                      PROD  [ Service A ] [ Service B ] [ Service C ]

Operating checklist

• keepalived health checks confirmed to detect a master failure and promote the standby within an acceptable window
• Both external and internal Elastic IPs reassign together on failover — tested by forcing a failover, not just assumed
• IAM user for the failover script scoped to Elastic IP association only, no broader EC2 permissions
• haproxy_connect_any SELinux boolean enabled on both nodes
• Certificate renewal hook confirmed to reload HAProxy on both nodes, not just the current master
• Config validated with -c before every reload, as a matter of habit rather than only after an incident