← tonia.ai

Distributed systems · security

Trust no network

A single computer has one kernel guarding its resources. A distributed system has many machines, owned and run by different people, talking over networks that anyone might be listening to. Securing it means proving who is on the other end of every message, keeping messages private and unaltered, deciding who may do what, and staying available when someone is actively trying to break it.

Why it's harder

  • Every message crosses untrusted ground. Anything sent between machines can be read, altered, delayed, replayed, or forged unless it is protected.
  • No single point of control. Each node enforces its own rules, so one weak node can undermine the rest.
  • More attack surface. Every service, port, credential, and dependency is a way in, and modern systems have thousands of each.
  • Mutual distrust. Nodes may belong to different organizations, and some may be compromised or malicious.

Two principles shape everything below. Saltzer and Schroeder's design principles (1975) still hold: least privilege, fail-safe defaults, complete mediation of every access, and an open design that doesn't depend on secrecy of the mechanism. And security is layered: no single control is enough, so defenses are stacked so that one failure doesn't expose everything.

PaperSaltzer & Schroeder, "The Protection of Information in Computer Systems," Proc. IEEE 63(9), 1975

Goals and threats

The core goals are confidentiality (only intended parties can read data), integrity (data can't be changed undetected), and availability (the service keeps working). Distributed systems add authenticity (knowing who sent a message) and sometimes non-repudiation (the sender can't later deny it).

Security protocols are analyzed against a strong attacker. In the Dolev–Yao model, the adversary controls the network: it can read, block, reorder, replay, and inject any message, but can't break the cryptography itself.

service A service B message attacker interceptread it modifychange it in transit fabricate / replayforge or resend old ones interruptblock or flood protect against: encryption · MACs and signatures · nonces and timestamps · redundancy and rate limits
ThreatExampleMain defense
EavesdroppingSniffing unencrypted traffic between servicesEncryption in transit
Man-in-the-middleAttacker poses as each side to the otherAuthenticated key exchange, certificates
ReplayResending a captured "transfer $100" requestNonces, timestamps, sequence numbers
SpoofingForging a sender's identityAuthentication, signatures
Denial of serviceFlooding a service so it can't respondRate limiting, filtering, spare capacity
SybilOne attacker creating many fake identitiesAdmission control, costly identities
Insider / compromised nodeA valid node behaving maliciouslyLeast privilege, audit, Byzantine tolerance
Supply chainMalicious code in a trusted update or dependencySigned builds, provenance, isolation

PaperDolev & Yao, "On the Security of Public Key Protocols," IEEE Trans. Information Theory 29(2), 1983
ReadSTRIDE threat categories (Microsoft) · OWASP Top 10

Cryptographic tools

Symmetric encryption
One shared key encrypts and decrypts (AES, ChaCha20). Fast, but both sides must already share the key.
Public-key encryption
A public key encrypts; only the matching private key decrypts (RSA, elliptic curves). No shared secret needed, but much slower.
Hash functions
A fixed-size fingerprint of any data (SHA-256). Infeasible to reverse or to find two inputs with the same hash.
MACs
A keyed hash (HMAC) proves a message came from someone holding the shared key and wasn't altered.
Digital signatures
Signed with a private key, verified with the public key. Anyone can check them, so they also give non-repudiation.
Authenticated encryption
Encryption and integrity in one step (AES-GCM, ChaCha20-Poly1305), the modern default.

Real systems combine them in hybrid encryption: public-key operations protect a fresh random symmetric key, and that key protects the bulk data.

SenderRecipient large data random key k AES-GCM with k encrypt k withB's public key ciphertextwrapped key decrypt with B'sprivate key → k AES-GCM with k slow public-key math runs once on 32 bytes; fast symmetric math handles the data

Cryptography fails far more often through misuse than through broken algorithms: reused nonces, unauthenticated encryption, homemade protocols, and keys stored next to the data they protect. Longer keys don't fix any of those. Use well-reviewed libraries and standard protocols instead of assembling primitives by hand.

PapersDiffie & Hellman, "New Directions in Cryptography," IEEE Trans. Information Theory 22(6), 1976 · Rivest, Shamir & Adleman, "A Method for Obtaining Digital Signatures and Public-Key Cryptosystems," CACM 21(2), 1978
TryPython cryptography library · CryptoHack (hands-on challenges)

Agreeing on a key

Diffie–Hellman key exchange lets two parties who have never met agree on a shared secret over a public channel. Each picks a private number, sends a public value derived from it, and combines the other's public value with its own private number. An eavesdropper sees only the public values. Here is a toy example with tiny numbers; real systems use elliptic curves with 256-bit keys.

public: prime p = 23, generator g = 5 AliceBob secret a = 6 secret b = 15 A = 5⁶ mod 23 = 8B = 5¹⁵ mod 23 = 19 A = 8 →← B = 19 19⁶ mod 238¹⁵ mod 23 shared = 2 shared = 2 eavesdropper sees 23, 5, 8, 19and still can't compute 2

Plain Diffie–Hellman has a gap: nothing proves who sent each public value. An active attacker can run a separate exchange with each side and relay between them, which is a man-in-the-middle attack. Real protocols therefore authenticate the exchange, usually by having the server sign its public value with a key vouched for by a certificate. Using fresh key pairs for each session also gives forward secrecy: stealing a server's long-term key later doesn't expose past sessions.

Certificates and PKI

How does a client know a public key really belongs to bank.example? A certificate binds a name to a public key and is signed by a certificate authority (CA). Clients ship with a small set of trusted root CAs and verify a chain of signatures from the server's certificate up to one of them.

root CAself-signedin the trust store intermediate CAsigned by root bank.examplepublic key, expirysigned by intermediate signssigns client checks each signature, the name, the dates, and revocation trust in bank.example rests entirely on trust in the root

The weak point is the CAs themselves: any trusted CA can issue a certificate for any name. Compromised or careless CAs have issued fraudulent certificates for major sites. Certificate Transparency responds by requiring every issued certificate to be recorded in public, append-only logs, so domain owners can spot certificates they never requested. Automated CAs such as Let's Encrypt made short-lived certificates free, which pushed most of the web to HTTPS.

PaperLaurie, "Certificate Transparency," CACM 57(10), 2014
Specs and toolsRFC 9162 (CT 2.0) · crt.sh (search CT logs for your domain) · How Let's Encrypt works

Secure channels: TLS

Transport Layer Security combines everything above into one protocol: an authenticated Diffie–Hellman exchange to agree on keys, certificates to prove identity, and authenticated encryption for the data. TLS 1.3 completes the handshake in one round trip and removed older, weaker options.

clientserver ClientHello + key share ServerHello + key share both now derive handshake keys {certificate, signature, Finished} {Finished} + application data { } encrypted one round trip before the client can send data

On the public web, only the server usually proves its identity, and the user logs in afterward. Between services inside a system, mutual TLS (mTLS) has both sides present certificates, so every service call is authenticated in both directions. Service meshes automate this by issuing short-lived certificates to every workload and rotating them continuously. TLS 1.3 was designed alongside formal analysis, which found and fixed flaws before the standard was published.

Spec and paperRFC 8446 (TLS 1.3) · Cremers et al., "A Comprehensive Symbolic Analysis of TLS 1.3," ACM CCS 2017
TryInspect TLS in Wireshark · SSL Labs server test · Istio mTLS

Authentication protocols

Authentication proves identity. Between machines, it is done with protocols that show possession of a secret without ever sending it. Two classic ideas run through all of them. Nonces are fresh random challenges that make each run unique, so recorded messages can't be replayed. A trusted third party vouches for identities, so every pair of machines doesn't need its own shared key.

Kerberos, built at MIT from the Needham–Schroeder protocol, is still the backbone of enterprise login, including Windows Active Directory. A user authenticates once and then receives time-limited tickets for each service.

key distribution center authentication server ticket-granting server client file service 1 log in as ana 2 TGT 3 TGT + "file service" 4 service ticket 5 service ticket + authenticator 6 proof of server identity
The password is used once, locally, to unlock the ticket-granting ticket (TGT). Tickets expire after hours, and each request carries a fresh timestamp so captured messages can't be replayed.

Protocol design is notoriously subtle. The Needham–Schroeder public-key protocol was used for seventeen years before Gavin Lowe found a man-in-the-middle attack on it in 1995 using a model checker. Today, important protocols are checked with formal tools before deployment.

PapersNeedham & Schroeder, "Using Encryption for Authentication in Large Networks of Computers," CACM 21(12), 1978 · Neuman & Ts'o, "Kerberos: An Authentication Service for Computer Networks," IEEE Communications 32(9), 1994 · Lowe, "Breaking and Fixing the Needham–Schroeder Public-Key Protocol Using FDR," TACAS 1996 · Lampson et al., "Authentication in Distributed Systems: Theory and Practice," ACM TOCS 10(4), 1992
DocsMIT Kerberos documentation

Tokens and federation

On the web and between microservices, identity usually travels as a token: a signed statement that a user or service authenticated and holds certain permissions. A service verifies the signature locally instead of calling a central server on every request.

OAuth 2.0
Delegated authorization: a user lets an app act on their behalf with limited scope, without giving it their password.
OpenID Connect
An identity layer on OAuth 2.0 that adds a signed ID token saying who the user is: "Sign in with…".
JWT
A compact, signed JSON format for tokens: header, claims (who, what, expiry), and signature.
Multi-factor authentication
Combines something you know, have, and are. It sharply reduces credential theft but doesn't stop phishing proxies, session hijacking, or push-fatigue attacks.

Self-contained tokens trade revocation for speed: a stolen token stays valid until it expires. The usual answer is short-lived access tokens plus refresh tokens that the issuer can revoke. Common mistakes include accepting unsigned tokens, failing to check the audience and expiry, and storing tokens where page scripts can read them.

SpecsRFC 6749 (OAuth 2.0) · OpenID Connect Core · RFC 7519 (JWT)
TryJWT introduction and decoder

Access control

Authorization decides what an authenticated identity may do. Conceptually, it is an access matrix of subjects and objects. Systems store that matrix in one of two ways.

orders DBpayments APIlogs webbillinganalyticsadmin read, write—append readchargeappend read—read allrefundread column = ACLstored with the object row = capabilities
ModelDecision based onStrengthWeakness
Access control listsPer-object list of who may do whatEasy to see and revoke access to one objectHard to see everything one subject can reach
CapabilitiesUnforgeable tokens the subject presentsEasy to delegate; no central lookupRevocation is hard once tokens spread
Role-based (RBAC)Roles assigned to users; permissions to rolesScales to large organizationsRole explosion as exceptions pile up
Attribute-based (ABAC)Rules over user, resource, and context attributesFine-grained, context-aware ("only on-call, only during incidents")Policies can be hard to audit

In distributed systems, authorization is increasingly expressed as policy as code: a central policy engine evaluates rules, and every service asks it, or a local copy of it, before acting. Whatever the model, the principle of least privilege holds: grant each identity only what it needs, for only as long as it needs it.

Papers and standardsDennis & Van Horn, "Programming Semantics for Multiprogrammed Computations," CACM 9(3), 1966 (origin of capabilities) · Sandhu et al., "Role-Based Access Control Models," IEEE Computer 29(2), 1996 · NIST SP 800-162, Guide to ABAC
ToolsOpen Policy Agent · Kubernetes RBAC

Zero trust

The traditional model is a perimeter: a hard firewall outside, and trusted traffic inside. It fails once anything inside is compromised, because attackers then move laterally unchallenged. Firewalls also can't see inside encrypted traffic or stop a legitimate user misusing their access.

PerimeterZero trust firewall one breached host reaches everything every call checked: identity, device, policy policy check

Zero trust drops the idea of a trusted network. Every request, even inside the datacenter, must prove the identity of both the user and the workload, come from a healthy device, and pass a policy check. Traffic is encrypted everywhere. Google's BeyondCorp, built after a 2009 intrusion, showed this works at scale: employees reach internal apps from any network without a VPN. Workload identity standards such as SPIFFE give every service a verifiable identity independent of its network address.

Papers and standardsWard & Beyer, "BeyondCorp: A New Approach to Enterprise Security," ;login: 39(6), 2014 · NIST SP 800-207, Zero Trust Architecture · SPIFFE overview

Attacks on the system itself

Some attacks target the distributed system's structure instead of individual messages.

  • Distributed denial of service. Botnets of compromised devices flood a target with traffic. The 2016 Mirai botnet, built from poorly secured cameras and routers, took down major DNS infrastructure. Defenses include absorbing traffic across a large network, filtering upstream, and rate limiting.
  • Sybil attacks. In open systems where joining is free, one attacker can create thousands of identities and outvote honest nodes. Douceur proved that without a central authority vouching for identities, Sybil attacks can't be fully prevented. That is why open blockchains make identities costly through proof of work or stake.
  • Byzantine nodes. A compromised member that lies can be tolerated with Byzantine fault-tolerant protocols, which need 3f + 1 replicas to survive f malicious ones.
  • Cryptography that ages. Large quantum computers would break today's public-key algorithms. Data captured now could be decrypted later, so systems are migrating to post-quantum algorithms. NIST standardized the first of them in 2024 (ML-KEM for key exchange, ML-DSA for signatures).

Papers and standardsAntonakakis et al., "Understanding the Mirai Botnet," USENIX Security 2017 · Douceur, "The Sybil Attack," IPTPS 2002 · Castro & Liskov, "Practical Byzantine Fault Tolerance," OSDI 1999 · FIPS 203 (ML-KEM) · FIPS 204 (ML-DSA)

In practice

Heartbleed, 2014

implementation bug

A missing bounds check in OpenSSL's heartbeat extension let anyone read up to 64 KB of a server's memory per request, including private keys and passwords. The cryptography was sound; the code around it was not. A large measurement study tracked how slowly the internet patched and revoked exposed certificates.

Durumeric et al., "The Matter of Heartbleed," IMC 2014

SolarWinds, 2020

supply chaintrusted updates

Attackers compromised a vendor's build system and inserted a backdoor into a signed software update that thousands of organizations installed. The signature was valid because the attackers had subverted the process that produced it. The case drove adoption of reproducible builds, build provenance, and limits on what monitoring software is allowed to reach.

CISA advisory AA20-352A

Cloud misconfiguration

shared responsibility

Many large data exposures involve no exploit at all, just storage buckets or databases left publicly readable, or service roles granted far more access than they need. Cloud providers secure the infrastructure; customers remain responsible for their own configurations, identities, and data.

Tools and reading

Try it

  • WiresharkCapture the same request over HTTP and HTTPS and compare what an eavesdropper sees.
  • OpenSSL command lineGenerate keys, create a small CA, and issue and verify certificates.
  • Python cryptographyImplement hybrid encryption and signatures with a vetted library.
  • Open Policy AgentWrite RBAC and ABAC policies as code and test them.
  • HashiCorp VaultIssue short-lived secrets and certificates instead of static passwords.
  • crt.shSee every certificate ever logged for a domain you know.

Read it