This documentation describes the ShardPost code base; the sections below carry the version and date they were written for. The architecture describes mechanisms that exist; concrete operational results carry a date and a piece of evidence. Goals are marked as goals. The product name is ShardPost, since 0.13.0 also in every operational identifier.
Reading paths
New to the project: Overview → Architecture → Runtime and UML → Developer guide.
Storage change: Storage guide → Consistency and data model → Architecture decisions → Tests and limits.
Operations: Configuration → Operations and releases → Logs, metrics, and traces. The day-to-day commands are documented as manual pages.
| Document | Contents |
|---|---|
| Architecture | System boundaries, components, deployment, impact of outages |
| Protocols, features and RFCs | Per-protocol reference: capabilities, commands, methods, limits, deviations, RFC index |
| Runtime and UML | Go interfaces, login, delivery, IMAP states |
| Storage guide | How ShardPost uses Ceph, built up from the basics: Ceph, the pools, one account's objects, walkthroughs, programs, operator playbook, monitoring, sizing |
| RADOS storage | Object layout, CAS, shards, intents, references, repair |
| Developer guide | Code map, build, tests, change paths, vendor fork |
| Quality | Test levels, conformance limits, current and historical evidence |
| Stage 8: operational administration | Planned: shardpostctl, OpenAPI, mailboxes, ACL, quota, FTS, repairs, and migration |
| Operations | Dependencies, safe diagnosis, release and rollback, running without Kubernetes |
| Bench | Bench pools, shardpost bench populate/measure, what an account costs in Ceph |
| Configuration | Viper, environment variables, secret files, log levels |
| Sieve and learning | Script execution, external programs, learning queue |
| Quota | Grace, mailbox rules, warning programs, over flag |
| Admin API and LDA | OpenAPI contract, shardpostctl, roles, local delivery |
| JMAP | Account state, session resource, API endpoint, blobs, what is built and what is not |
| JMAP by example | A recorded send and receive through the Director, as JSON |
| Manual pages | shardpostctl command by command, with examples |
| Observability | What the metrics mean, sampling, privacy, operational logs |
| Architecture decisions | Reasons for and consequences of the central design decisions |
| Project name | Candidates and a compatible rename |
| Diagrams | Mermaid sources, SVGs, and reproducible rendering |
Evidence and history
- ShardPost 0.51.0: online reshard and rebuild of the full-text index without account lock, night window (stage B14i).
- ShardPost 0.52.0: JMAP push partitions, job runner pods, online quota recount, mailbox lock (stages B16, B17, B18).
- Stage B18 implementation: push partitions and job runner pods as built, deviations from the plan.
- ShardPost 0.50.0: identifier dictionary filter and attached-header text preparation, off by default (stage B14h).
- ShardPost 0.49.0: separate full-text caps, worker no-op fix, parallel conversions, fts stats; offline measurement for B14h (stage B14g).
- Full-text dictionary V0: the largest account resharded to 128 dictionary objects, large-omap warning cleared.
- System pool cutover on croessner: store-wide namespaces moved to shardpost-system with 0.48.0, about two minutes of downtime.
- ShardPost 0.47.0: system pool routing, fence and mark, registration roles, hotspot fixes (stage B11a).
- ShardPost 0.46.0: full-text layout 3 is the default (with the cluster mark), fts_dict_convert legacy (stage B14c).
- ShardPost 0.45.1: layout 3 switched on for all accounts; JMAP 503 with Retry-After for locked accounts.
- ShardPost 0.45.0: full-text layout 3, off by default (stage B14b); 24 instead of 226 operations per delivery in the bench.
- ShardPost 0.44.0: one read per posting shard, CAS backoff, bench probes (stage B14a).
- Scaling stage R4 measurement: what an account costs in Ceph (bench of 1000 accounts); the full-text index makes 92 % of the operations per delivery.
- ShardPost 0.43.0: delayed JMAP send (scaling stage R5), bench tool for stage R4.
- ShardPost 0.42.0: intents, scrub and purge without per-account listings, sweep (scaling stage R3).
- ShardPost 0.41.0: auth cache with cluster-wide flush, configurable Nauthilus client, JMAP wait queue and limits.
- ShardPost 0.40.0: batch over all accounts, partitioned scrub with skip-if-idle, IMAP watch hub (scaling stage R2b).
- ShardPost 0.39.0: no pool listing in running paths, per-account limits (scaling stage R2a).
- ShardPost 0.38.0: job queue and expiry without listing, key rotation of the admin store (scaling stage R1b).
- ShardPost 0.37.0: account directory (scaling stage R1a), ESORT.
- ShardPost 0.36.0/0.36.1: the gaps of the protocol reference closed, imaptest without failures.
- ShardPost 0.35.2: protocol fixes from the documentation audit.
- JMAP Core 0.27.0: account-wide state, JMAP Core, release evidence.
- ShardPost 0.13.0: lint cleanup, namespace/pool/identity move, current rollout.
- ShardPost 0.12.0: rename and compatibility.
- Logging 0.11.3: logging extension and the protocol evidence of the time.
- Sieve fixes up to 0.11.2.
- External programs.
- IMAPSieve and PROXY protocol.
- Conformance results: versioned results, no blanket green.
- Review round 1, 2, 3, 4, 5, 6: historical findings and changes.
- Earlier architecture, earlier README, build-out history.
Historical documents are kept as rationale and evidence. New implementations follow the current code map and the interfaces that are actually implemented. When a flow changes, its description, the Mermaid source, and the rendered SVG are updated together.
Rendered from docs/README.md in the ShardPost repository.