ShardPost
How it works

From the app on the phone
to the object on the disk.

A ShardPost mail system has four layers: clients, the authentication and routing in front, the ShardPost replicas, and Ceph underneath. Select any part of the diagram to see what it does.

OIDC sign-in JMAP LMTP auth · routing TLS+PROXY gRPC librados Web Push CLIENT APP JMapper Mail iPhone · iPad · Mac JMAP + OIDC MAIL CLIENTS IMAP · POP3 ManageSieve INBOUND MAIL MTA e.g. Postfix, queues SMTP NOTIFICATIONS Web Push the client's push service, e.g. JMapper's gateway IDENTITY & POLICY Nauthilus gRPC AuthService OIDC provider · MFA token introspection LDAP · Lua policies PROTOCOL DIRECTOR nauthilus-director authenticates sessions routes by mail shard proxies to a backend IMAP · POP3 · Sieve · LMTP · JMAP MAIL SERVER ShardPost IMAP POP3 JMAP LMTP Sieve IMAPSieve replica 1same pools replica 2same pools replica nsame pools push64 partitions runnersjobs, learning STORAGE Ceph RADOS index pool replicated · omap mailboxes, entries data pool erasure coded message blobs system pool optional store-wide state a namespace per account OUTBOUND SMTP relay Sieve actions, JMAP sending TELEMETRY Monitoring Prometheus, OTLP traces, logs ADMINISTRATION shardpostctl admin API (OpenAPI) Unix socket or mTLS
JMAP and sign-in mail protocols authentication storage outbound and push

Select a part of the diagram to read what it does. The layout follows the reference deployment; Nauthilus is the authentication backend, nauthilus-director the optional front door.

ShardPost

Select a part of the diagram to read what it does.

Step by step

What happens when …

Three everyday paths through the stack, written down from the storage guide and the protocol reference.

… a JMapper user opens the inbox

JMapper on an iPhone, Nauthilus as the OpenID Connect provider, ShardPost serving JMAP.

  1. Discovery. JMapper discovers the JMAP endpoint and the OpenID Connect issuer of the mail domain.
  2. Sign-in. It registers itself with Nauthilus and signs the user in through the system browser with PKCE, including any second factor. It keeps only the issued tokens.
  3. Request. Every JMAP request carries the bearer token. nauthilus-director checks it by introspection, finds the account's mail shard and forwards the request with the PROXY protocol.
  4. Authentication. ShardPost checks the token through Nauthilus (RFC 7662) and its auth cache and gets the canonical account.
  5. Reading. The account, its mailbox list and the INBOX state cost four RADOS reads. Envelopes and previews come from the index entries; the message bytes are read only when the body is opened.
  6. Staying current. Email/changes answers from the account change log. When new mail arrives, a push worker sends an encrypted Web Push notification to iPhone and iPad.

… a mail arrives

The MTA in front hands the message over with LMTP.

  1. Recipient. The LMTP session asks the auth cache, and on a miss Nauthilus, for the canonical account. An unknown recipient is refused before the message is sent.
  2. Sieve. The account's active script and the global scripts decide: keep, fileinto, redirect, vacation.
  3. Reserve. One compare-and-swap checks the quota, adds the size and records an intent that a blob is about to be written. Over quota, LMTP answers 552 5.2.2.
  4. Write and index. The blob goes into the data pool, then its text into the full-text index.
  5. Link. A compare-and-swap on the mailbox allocates the next UID and MODSEQ and writes the entry; the account log records the change.
  6. Wake up. A notify on the mailbox and the account wakes IDLE and NOTIFY sessions and JMAP push workers on every server. LMTP answers 250.

The full walkthrough in the storage guide →

… someone logs in with IMAP

A login is mostly not a Ceph event.

  1. Front door. nauthilus-director takes the connection, authenticates the session through Nauthilus and picks a healthy ShardPost backend.
  2. Handover. It connects to ShardPost with TLS and a PROXY header, so ShardPost sees the client's real address. In the reference deployment the Director logs in on the user's behalf as a master user (<account>*nauthilus-director; name and separator are configurable).
  3. Check. ShardPost asks its auth cache, which keeps a successful login for 30 seconds by default, and on a miss Nauthilus, with client address, protocol and TLS state.
  4. Session. The session opens the account's RADOS namespace. Everything it reads afterwards comes from the index and data pools.

Authentication in the protocol reference →

… two servers write the same mailbox

Two deliveries to the same INBOX, on two replicas, at the same moment.

  1. Read. Both read UIDNEXT 43 together with the object version.
  2. Write with a condition. Both write “only if the object is still at that version”. RADOS lets exactly one of them win.
  3. Retry. The other write fails; that server reads again, gets 44 and writes 44. No UID is ever given twice.
  4. Notify. Each successful write rings the mailbox's watch, so the IDLE sessions on both servers see both messages.

Many servers, no central database →

Coordination

Coordination without a database server.

RADOS makes the change of one object atomic. Everything else ShardPost needs to run many replicas against the same accounts is built on that.

Compare-and-swap

Read the keys and the object version, compute, write only if the version is unchanged, otherwise try again. Every shared counter (UIDs, MODSEQ, quota) works this way.

Announce, write, record

A change across two objects is announced first and recorded last. Readers never move past an announcement without a record, so a JMAP state or an IMAP MODSEQ never skips a change.

Watch and notify

A doorbell, not a letter: the writer rings, the watcher looks for itself. A lost notification loses nothing, because the change log stays the source of truth.

Locks and leases

Locks in the OSD pick a single worker; leases in job records let another runner take over a job from its last checkpoint. The compare-and-swap on the data stays the final fence.

Intents

Before a write that could leave debris, ShardPost records an intent and removes it afterwards. A crash turns into a to-do list for the scrub, which needs no pool listing to find it.

Process-local sessions

Connections, parsers and caches belong to one process; durable state belongs to RADOS. A client that moves to another replica needs only a new connection.

Boundaries

Who does what, and what happens when it fails.

Responsibilities

  • ShardPost: IMAP, POP3, JMAP, LMTP, ManageSieve, Sieve and the mail store; no public SMTP and no user management.
  • Nauthilus: login and the resolution of an address to the canonical account; no mail content.
  • nauthilus-director: public protocol access and backend routing, client address via PROXY.
  • Ceph: persistence, replication and erasure coding; no transaction across mail objects.
  • SMTP relay: redirect, vacation, notifications and JMAP sending.

Failures and their visible effect

  • Nauthilus unreachable: new logins fail temporarily and are reported as unavailable, never as a wrong password.
  • Ceph unreachable: store operations fail and readiness turns red; LMTP answers with a temporary error, so the MTA retries.
  • A replica stops: its sessions end after a drain; clients reconnect to another one.
  • A notification is lost: updates arrive later; the change log remains authoritative.

From the architecture overview. The architecture decisions explain why it is built this way, and what each decision costs.

Go deeper

The storage guide starts with Ceph in five minutes.

It builds the picture from the basics: Ceph, the pools, one account's objects, walkthroughs, the programs, the operator's job, monitoring and sizing.

Read the storage guide All documentation