01Where protection lives
Every secure protocol answers one design question: at which layer, and between which two endpoints, is data protected? Lower layers protect more traffic transparently; higher layers protect less but understand it better and can carry it farther.
- Wireshark sample captures — real TLS, ESP, and IKE traffic to dissect
- Holz, R. et al. (2016). TLS in the wild: an Internet-wide analysis of TLS-based protocols for electronic communication. NDSS.
02TLS as a protocol architecture
TLS sits between an application and TCP (or inside QUIC over UDP) and is really two cooperating sub-protocols. The handshake authenticates the server (and optionally the client), negotiates algorithms, and derives keys. The record layer then fragments application data, protects each record with an AEAD cipher, and numbers them so reordering or replay is detected.
Two deployment styles exist. Implicit TLS starts encrypted on a dedicated port (HTTPS on 443, IMAPS on 993). STARTTLS begins in plaintext and upgrades, which leaves a window where an active attacker can strip the upgrade offer and force cleartext.
Version history in one line
SSL 2.0 and 3.0 are broken; TLS 1.0 and 1.1 are formally deprecated; TLS 1.2 is acceptable when configured with AEAD suites; TLS 1.3 removes the legacy options entirely. "SSL" survives only as a marketing word.
- testssl.sh · SSL Labs — audit a server's versions, suites, and certificate
- NIST SP 800-52 Rev. 2 — TLS implementation guidelines · RFC 8996 — deprecating TLS 1.0/1.1
- Bhargavan, K. et al. (2014). Triple handshakes and cookie cutters: breaking and fixing authentication over TLS. IEEE S&P.
03IPsec: protecting packets
IPsec works at the network layer, below every application, so it secures all IP traffic between two endpoints without any software changes. It has two protection protocols and two modes, and the combination decides what an observer can still see.
AH Authentication Header
Integrity and origin authentication, including most of the outer IP header. No encryption. Also breaks through NAT. Rarely used today.
ESP Encapsulating Security Payload
Encryption plus integrity of the payload. With AEAD ciphers it does everything most deployments need; it is the de facto default.
- RFC 4301 — IPsec architecture · RFC 4303 — ESP · RFC 4302 — AH
- NIST SP 800-77 Rev. 1 — Guide to IPsec VPNs
- strongSwan documentation — open-source IPsec for lab tunnels
- Wireshark ESP decryption — supply keys to see inside a capture
04IKEv2 and security associations
ESP needs keys and agreed algorithms before the first packet. A security association (SA) is that agreement: one-directional, identified by an SPI number carried in each ESP header, and stored in each peer's database. IKEv2 is the protocol that creates SAs automatically, authenticates peers, and rekeys them before they expire.
Authentication choices matter: pre-shared keys are simple but shared across many devices and vulnerable to offline guessing if weak; certificates scale and can be revoked individually. A newer, smaller alternative, WireGuard, drops negotiation entirely and uses one fixed modern suite, trading agility for a codebase small enough to audit and formally verify.
- RFC 7296 — IKEv2 · WireGuard
- Felsch, D. et al. (2018). The dangers of key reuse: practical attacks on IPsec IKE. USENIX Security.
- Donenfeld, J. (2017). WireGuard: next generation kernel network tunnel. NDSS.
05Securing email
Email was designed in an era of trusted hosts and still relays messages through several servers that store and forward them. That makes "secure email" two separate problems: protecting each hop, and protecting the message itself.
Transport layer
STARTTLS between servers is opportunistic and strippable. MTA-STS and DANE let a domain demand authenticated TLS.
Sender authenticity
SPF lists allowed sending IPs, DKIM signs messages per domain, DMARC tells receivers what to do when both fail. These fight spoofing, not snooping.
End to end
S/MIME and OpenPGP encrypt to the recipient's public key and sign with the sender's private key. Strong, but notoriously hard to use.
- NIST SP 800-177 Rev. 1 — Trustworthy Email
- internet.nl — test a domain's STARTTLS, DANE, SPF, DKIM, and DMARC
- Gpg4win · GnuPG documentation
- RFC 8551 — S/MIME 4.0 · RFC 9580 — OpenPGP · RFC 8461 — MTA-STS · RFC 7489 — DMARC
- Durumeric, Z. et al. (2015). Neither snow nor rain nor MITM… an empirical analysis of email delivery security. ACM IMC.
06Two models of trust
Every public-key protocol must answer "how do I know this key belongs to that name?" The two email standards answer it in opposite ways, and the same split appears elsewhere (TLS uses hierarchy; SSH defaults to trust-on-first-use).
- Whitten, A. & Tygar, J. D. (1999). Why Johnny can't encrypt: a usability evaluation of PGP 5.0. USENIX Security.
- Poddebniak, D. et al. (2018). Efail: breaking S/MIME and OpenPGP email encryption using exfiltration channels. USENIX Security.
07Side-by-side comparison
| TLS | IPsec (ESP) | S/MIME / OpenPGP | |
|---|---|---|---|
| Layer | Between app and transport | Network | Application (message) |
| Protects | One connection | All IP traffic between peers | One message, at rest too |
| Authentication | X.509 certs; usually server only | Certs or PSK; both peers | Certs or web of trust; sender signs |
| Confidentiality | Yes (AEAD) | Yes (AEAD) | Yes, end to end |
| Integrity | Per record | Per packet | Per message (signature) |
| Non-repudiation | No | No | Yes, with signatures |
| App changes? | Yes, app must use TLS | None | Mail client support |
| NAT / firewall friendly | Very | Needs NAT-T (UDP 4500) | Rides on email |
| Hides metadata? | Not IPs, sizes, often SNI | Tunnel mode hides inner IPs | Headers visible |
TLS and IPsec authenticate a channel; only message-level signatures prove who authored the content to a third party.
08Choosing a protocol
Start from the threat and the boundary you need, not from the product.
09Pitfalls and misconceptions
A protocol proven secure on paper is only as strong as its implementation and configuration.
Implementation bugs
OpenSSL's Heartbleed and repeated flaws in commercial VPN appliances show that widely deployed code is a single point of failure for millions of hosts.
Downgrade
Any protocol that negotiates can be talked down to its weakest allowed option. Disable legacy versions; don't merely prefer modern ones.
Misconfiguration
AH-only policies, weak PSKs, missing certificate validation, and permissive STARTTLS all pass a casual test while giving little protection.
- CISA KEV catalog — filter for VPN and gateway products
- GNS3 — virtual routers for building and breaking IPsec labs
—References
Peer-reviewed
- Bhargavan, K. et al. (2014). Triple handshakes and cookie cutters. IEEE S&P. doi:10.1109/SP.2014.14
- Donenfeld, J. (2017). WireGuard: next generation kernel network tunnel. NDSS. doi:10.14722/ndss.2017.23160
- Durumeric, Z. et al. (2015). Neither snow nor rain nor MITM. ACM IMC. doi:10.1145/2815675.2815695
- Felsch, D. et al. (2018). The dangers of key reuse: practical attacks on IPsec IKE. USENIX Security. usenix.org
- Holz, R. et al. (2016). TLS in the wild. NDSS. doi:10.14722/ndss.2016.23055
- Poddebniak, D. et al. (2018). Efail. USENIX Security. usenix.org
- Whitten, A. & Tygar, J. D. (1999). Why Johnny can't encrypt. USENIX Security. usenix.org