Distributed systems · security
Trust no network
A single computer has one kernel guarding its resources. A distributed system has many machines, owned and run by different people, talking over networks that anyone might be listening to. Securing it means proving who is on the other end of every message, keeping messages private and unaltered, deciding who may do what, and staying available when someone is actively trying to break it.
Why it's harder
- Every message crosses untrusted ground. Anything sent between machines can be read, altered, delayed, replayed, or forged unless it is protected.
- No single point of control. Each node enforces its own rules, so one weak node can undermine the rest.
- More attack surface. Every service, port, credential, and dependency is a way in, and modern systems have thousands of each.
- Mutual distrust. Nodes may belong to different organizations, and some may be compromised or malicious.
Two principles shape everything below. Saltzer and Schroeder's design principles (1975) still hold: least privilege, fail-safe defaults, complete mediation of every access, and an open design that doesn't depend on secrecy of the mechanism. And security is layered: no single control is enough, so defenses are stacked so that one failure doesn't expose everything.
PaperSaltzer & Schroeder, "The Protection of Information in Computer Systems," Proc. IEEE 63(9), 1975
Goals and threats
The core goals are confidentiality (only intended parties can read data), integrity (data can't be changed undetected), and availability (the service keeps working). Distributed systems add authenticity (knowing who sent a message) and sometimes non-repudiation (the sender can't later deny it).
Security protocols are analyzed against a strong attacker. In the Dolev–Yao model, the adversary controls the network: it can read, block, reorder, replay, and inject any message, but can't break the cryptography itself.
| Threat | Example | Main defense |
|---|---|---|
| Eavesdropping | Sniffing unencrypted traffic between services | Encryption in transit |
| Man-in-the-middle | Attacker poses as each side to the other | Authenticated key exchange, certificates |
| Replay | Resending a captured "transfer $100" request | Nonces, timestamps, sequence numbers |
| Spoofing | Forging a sender's identity | Authentication, signatures |
| Denial of service | Flooding a service so it can't respond | Rate limiting, filtering, spare capacity |
| Sybil | One attacker creating many fake identities | Admission control, costly identities |
| Insider / compromised node | A valid node behaving maliciously | Least privilege, audit, Byzantine tolerance |
| Supply chain | Malicious code in a trusted update or dependency | Signed builds, provenance, isolation |
PaperDolev & Yao, "On the Security of Public Key Protocols," IEEE Trans. Information Theory 29(2), 1983
ReadSTRIDE threat categories (Microsoft) · OWASP Top 10
Cryptographic tools
- Symmetric encryption
- One shared key encrypts and decrypts (AES, ChaCha20). Fast, but both sides must already share the key.
- Public-key encryption
- A public key encrypts; only the matching private key decrypts (RSA, elliptic curves). No shared secret needed, but much slower.
- Hash functions
- A fixed-size fingerprint of any data (SHA-256). Infeasible to reverse or to find two inputs with the same hash.
- MACs
- A keyed hash (HMAC) proves a message came from someone holding the shared key and wasn't altered.
- Digital signatures
- Signed with a private key, verified with the public key. Anyone can check them, so they also give non-repudiation.
- Authenticated encryption
- Encryption and integrity in one step (AES-GCM, ChaCha20-Poly1305), the modern default.
Real systems combine them in hybrid encryption: public-key operations protect a fresh random symmetric key, and that key protects the bulk data.
Cryptography fails far more often through misuse than through broken algorithms: reused nonces, unauthenticated encryption, homemade protocols, and keys stored next to the data they protect. Longer keys don't fix any of those. Use well-reviewed libraries and standard protocols instead of assembling primitives by hand.
PapersDiffie & Hellman, "New Directions in Cryptography," IEEE Trans. Information Theory 22(6), 1976 · Rivest, Shamir & Adleman, "A Method for Obtaining Digital Signatures and Public-Key Cryptosystems," CACM 21(2), 1978
TryPython cryptography library · CryptoHack (hands-on challenges)
Agreeing on a key
Diffie–Hellman key exchange lets two parties who have never met agree on a shared secret over a public channel. Each picks a private number, sends a public value derived from it, and combines the other's public value with its own private number. An eavesdropper sees only the public values. Here is a toy example with tiny numbers; real systems use elliptic curves with 256-bit keys.
Plain Diffie–Hellman has a gap: nothing proves who sent each public value. An active attacker can run a separate exchange with each side and relay between them, which is a man-in-the-middle attack. Real protocols therefore authenticate the exchange, usually by having the server sign its public value with a key vouched for by a certificate. Using fresh key pairs for each session also gives forward secrecy: stealing a server's long-term key later doesn't expose past sessions.
Certificates and PKI
How does a client know a public key really belongs to bank.example? A certificate binds a name to a public key and is signed by a certificate authority (CA). Clients ship with a small set of trusted root CAs and verify a chain of signatures from the server's certificate up to one of them.
The weak point is the CAs themselves: any trusted CA can issue a certificate for any name. Compromised or careless CAs have issued fraudulent certificates for major sites. Certificate Transparency responds by requiring every issued certificate to be recorded in public, append-only logs, so domain owners can spot certificates they never requested. Automated CAs such as Let's Encrypt made short-lived certificates free, which pushed most of the web to HTTPS.
PaperLaurie, "Certificate Transparency," CACM 57(10), 2014
Specs and toolsRFC 9162 (CT 2.0) · crt.sh (search CT logs for your domain) · How Let's Encrypt works
Secure channels: TLS
Transport Layer Security combines everything above into one protocol: an authenticated Diffie–Hellman exchange to agree on keys, certificates to prove identity, and authenticated encryption for the data. TLS 1.3 completes the handshake in one round trip and removed older, weaker options.
On the public web, only the server usually proves its identity, and the user logs in afterward. Between services inside a system, mutual TLS (mTLS) has both sides present certificates, so every service call is authenticated in both directions. Service meshes automate this by issuing short-lived certificates to every workload and rotating them continuously. TLS 1.3 was designed alongside formal analysis, which found and fixed flaws before the standard was published.
Spec and paperRFC 8446 (TLS 1.3) · Cremers et al., "A Comprehensive Symbolic Analysis of TLS 1.3," ACM CCS 2017
TryInspect TLS in Wireshark · SSL Labs server test · Istio mTLS
Authentication protocols
Authentication proves identity. Between machines, it is done with protocols that show possession of a secret without ever sending it. Two classic ideas run through all of them. Nonces are fresh random challenges that make each run unique, so recorded messages can't be replayed. A trusted third party vouches for identities, so every pair of machines doesn't need its own shared key.
Kerberos, built at MIT from the Needham–Schroeder protocol, is still the backbone of enterprise login, including Windows Active Directory. A user authenticates once and then receives time-limited tickets for each service.
Protocol design is notoriously subtle. The Needham–Schroeder public-key protocol was used for seventeen years before Gavin Lowe found a man-in-the-middle attack on it in 1995 using a model checker. Today, important protocols are checked with formal tools before deployment.
PapersNeedham & Schroeder, "Using Encryption for Authentication in Large Networks of Computers," CACM 21(12), 1978 · Neuman & Ts'o, "Kerberos: An Authentication Service for Computer Networks," IEEE Communications 32(9), 1994 · Lowe, "Breaking and Fixing the Needham–Schroeder Public-Key Protocol Using FDR," TACAS 1996 · Lampson et al., "Authentication in Distributed Systems: Theory and Practice," ACM TOCS 10(4), 1992
DocsMIT Kerberos documentation
Tokens and federation
On the web and between microservices, identity usually travels as a token: a signed statement that a user or service authenticated and holds certain permissions. A service verifies the signature locally instead of calling a central server on every request.
- OAuth 2.0
- Delegated authorization: a user lets an app act on their behalf with limited scope, without giving it their password.
- OpenID Connect
- An identity layer on OAuth 2.0 that adds a signed ID token saying who the user is: "Sign in with…".
- JWT
- A compact, signed JSON format for tokens: header, claims (who, what, expiry), and signature.
- Multi-factor authentication
- Combines something you know, have, and are. It sharply reduces credential theft but doesn't stop phishing proxies, session hijacking, or push-fatigue attacks.
Self-contained tokens trade revocation for speed: a stolen token stays valid until it expires. The usual answer is short-lived access tokens plus refresh tokens that the issuer can revoke. Common mistakes include accepting unsigned tokens, failing to check the audience and expiry, and storing tokens where page scripts can read them.
SpecsRFC 6749 (OAuth 2.0) · OpenID Connect Core · RFC 7519 (JWT)
TryJWT introduction and decoder
Access control
Authorization decides what an authenticated identity may do. Conceptually, it is an access matrix of subjects and objects. Systems store that matrix in one of two ways.
| Model | Decision based on | Strength | Weakness |
|---|---|---|---|
| Access control lists | Per-object list of who may do what | Easy to see and revoke access to one object | Hard to see everything one subject can reach |
| Capabilities | Unforgeable tokens the subject presents | Easy to delegate; no central lookup | Revocation is hard once tokens spread |
| Role-based (RBAC) | Roles assigned to users; permissions to roles | Scales to large organizations | Role explosion as exceptions pile up |
| Attribute-based (ABAC) | Rules over user, resource, and context attributes | Fine-grained, context-aware ("only on-call, only during incidents") | Policies can be hard to audit |
In distributed systems, authorization is increasingly expressed as policy as code: a central policy engine evaluates rules, and every service asks it, or a local copy of it, before acting. Whatever the model, the principle of least privilege holds: grant each identity only what it needs, for only as long as it needs it.
Papers and standardsDennis & Van Horn, "Programming Semantics for Multiprogrammed Computations," CACM 9(3), 1966 (origin of capabilities) · Sandhu et al., "Role-Based Access Control Models," IEEE Computer 29(2), 1996 · NIST SP 800-162, Guide to ABAC
ToolsOpen Policy Agent · Kubernetes RBAC
Zero trust
The traditional model is a perimeter: a hard firewall outside, and trusted traffic inside. It fails once anything inside is compromised, because attackers then move laterally unchallenged. Firewalls also can't see inside encrypted traffic or stop a legitimate user misusing their access.
Zero trust drops the idea of a trusted network. Every request, even inside the datacenter, must prove the identity of both the user and the workload, come from a healthy device, and pass a policy check. Traffic is encrypted everywhere. Google's BeyondCorp, built after a 2009 intrusion, showed this works at scale: employees reach internal apps from any network without a VPN. Workload identity standards such as SPIFFE give every service a verifiable identity independent of its network address.
Papers and standardsWard & Beyer, "BeyondCorp: A New Approach to Enterprise Security," ;login: 39(6), 2014 · NIST SP 800-207, Zero Trust Architecture · SPIFFE overview
Attacks on the system itself
Some attacks target the distributed system's structure instead of individual messages.
- Distributed denial of service. Botnets of compromised devices flood a target with traffic. The 2016 Mirai botnet, built from poorly secured cameras and routers, took down major DNS infrastructure. Defenses include absorbing traffic across a large network, filtering upstream, and rate limiting.
- Sybil attacks. In open systems where joining is free, one attacker can create thousands of identities and outvote honest nodes. Douceur proved that without a central authority vouching for identities, Sybil attacks can't be fully prevented. That is why open blockchains make identities costly through proof of work or stake.
- Byzantine nodes. A compromised member that lies can be tolerated with Byzantine fault-tolerant protocols, which need 3f + 1 replicas to survive f malicious ones.
- Cryptography that ages. Large quantum computers would break today's public-key algorithms. Data captured now could be decrypted later, so systems are migrating to post-quantum algorithms. NIST standardized the first of them in 2024 (ML-KEM for key exchange, ML-DSA for signatures).
Papers and standardsAntonakakis et al., "Understanding the Mirai Botnet," USENIX Security 2017 · Douceur, "The Sybil Attack," IPTPS 2002 · Castro & Liskov, "Practical Byzantine Fault Tolerance," OSDI 1999 · FIPS 203 (ML-KEM) · FIPS 204 (ML-DSA)
In practice
Heartbleed, 2014
implementation bug
A missing bounds check in OpenSSL's heartbeat extension let anyone read up to 64 KB of a server's memory per request, including private keys and passwords. The cryptography was sound; the code around it was not. A large measurement study tracked how slowly the internet patched and revoked exposed certificates.
SolarWinds, 2020
supply chaintrusted updates
Attackers compromised a vendor's build system and inserted a backdoor into a signed software update that thousands of organizations installed. The signature was valid because the attackers had subverted the process that produced it. The case drove adoption of reproducible builds, build provenance, and limits on what monitoring software is allowed to reach.
Cloud misconfiguration
shared responsibility
Many large data exposures involve no exploit at all, just storage buckets or databases left publicly readable, or service roles granted far more access than they need. Cloud providers secure the infrastructure; customers remain responsible for their own configurations, identities, and data.
Tools and reading
Try it
- WiresharkCapture the same request over HTTP and HTTPS and compare what an eavesdropper sees.
- OpenSSL command lineGenerate keys, create a small CA, and issue and verify certificates.
- Python cryptographyImplement hybrid encryption and signatures with a vetted library.
- Open Policy AgentWrite RBAC and ABAC policies as code and test them.
- HashiCorp VaultIssue short-lived secrets and certificates instead of static passwords.
- crt.shSee every certificate ever logged for a domain you know.
Read it
- van Steen & Tanenbaum, Distributed SystemsFree textbook chapter on security.
- Saltzer & Schroeder, 1975Design principles still used today.
- Diffie & Hellman, 1976The paper that introduced public-key cryptography.
- Lampson et al., 1992A rigorous theory of authentication and delegation.
- Ward & Beyer, 2014Zero trust in production.
- Cremers et al., 2017Formal verification of a real protocol.