shardpostctl-job(1)

plan

Computes what a job of that kind would do to the account and stores the result. Nothing is changed. The plan carries a fingerprint of the state it was computed from, and it expires (jobs.plan_ttl_seconds). Role: the one the kind needs, see Kinds below.

plan -all

Plans a batch of the kind over every account of the account directory: index_check, scrub, fts_rebuild, attachment_backfill, fts_layout or quota_recalc. -classes selects the repair classes of a scrub as for one account; -skip-idle (scrub only) leaves out the accounts that did not change since their last batch scrub was complete and found nothing, for at most scrub.interval_days (7). The plan names the kind, its role, whether it locks the accounts, the parallelism and the directory it walks; its account is empty and its kind batch. apply runs it as one job: it walks the directory a page at a time and works on jobs.batch_parallel accounts at once (scrub.parallel for a scrub). The result counts the accounts done, skipped, gone (purged meanwhile), failed and with findings, and names the first 50 that failed or had findings; a batch with a failed account ends partially_failed. A kind that locks the account (fts_rebuild, fts_layout and quota_recalc unless online) takes the lock per account and waits jobs.settle_seconds before it writes, as a single job does: while it holds an account, the open IMAP and POP3 sessions of that account are disconnected at their next command and deliveries wait, so such a batch disconnects every account once, one after another. An account an import or another job holds is tried again at the end, up to three times a lease period apart, and then counted as failed. A scrub with -skip-idle does not look at the private \Seen marks of deleted public folders, nor at debris that servers before 0.42.0 left without an intent, for the accounts it skips; they wait until the account is scrubbed again, at most scrub.interval_days later. An account with an open intent (what a crash left) is never skipped. A cancel stops the batch between two accounts. A server that takes the batch over continues after the last page done.

A batch of fts_layout or fts_rebuild keeps the reshard window (fts.reshard_window, since 0.51.0) unless -anytime is given: it starts an account only inside the window; outside it the job shows "waiting for the reshard window" and gives itself back until the next window starts (the attempt does not count), then goes on from its checkpoint. An online run that does not fit into what is left of the window waits for the next one (the result lists it as later); one whose estimate is longer than the whole window fails the account with window_too_small. Online, the accounts are not locked; an account another run of which is alive is tried again at the end as a locked one, and one without a layout 3 index counts as unsupported (not as failed).

A batch is refused (exit 65) until the account directory has completed its backfill (shardpost index backfill accounts), since accounts from before the directory would be missed. Role: the one the kind needs, and the scope of all accounts: socket callers have it, mutual-TLS clients only with all_accounts in their configuration. An export of every account is no batch kind: the export streams one archive per request to the caller.

plan show

Prints a stored plan again, by its id: what it would do, when it was made and when it expires. Role: diagnose.

apply

Runs a stored plan, once. With -wait the command stays until the job ends and exits with its outcome; -idempotency-key makes a repeated call return the first job instead of starting a second one. Role: the one the kind needs.

check

Runs the read-only index check on an account without a plan, because it changes nothing. Role: diagnose.

job list

The jobs of the cluster, oldest first, with their state. Narrow it with -account, -state and -limit; -after continues after the last id of the previous page. Role: diagnose.

job show

One job: kind, account, state, progress, attempts, the server running it, and the error of a failed one. Role: diagnose.

job wait

Waits for a job to end and exits with its outcome, polling every -poll (2s by default). This is what a script uses instead of a loop of job show. Role: diagnose.

job cancel

Asks a job to stop. A job that has not started yet never starts; a running one stops at its next safe point, and what it has already done stays done. Role: the one the kind needs (for a batch the one of the kind it runs, and the scope of all accounts).

Kinds

index_check

Role diagnose. Checks the index and reports findings. Repairs nothing. This is what check runs.

scrub

Role repair. Checks and repairs. -classes selects what may be repaired: deliveries, moves, retired, uncovered, entries, refs, orphans, unknown, quota. Without the flag every class is repaired. Where the scrub looks for what nothing points at is scrub.orphan_source in the servers' configuration: listing lists the account's namespace in both pools, shadow (the default) does that and also works out what the intents name, reporting the differences in the field Shadow of the result, intents reads the intents only. With intents, orphans covers the blobs of open intents and unknown the objects of abandoned creations and unfinished deletions, the marks of deleted public folders and the METADATA of deleted mailboxes. The result counts, besides the findings, the intents older than scrub.intent_grace_seconds (IntentsOpen) and those removed (IntentsDropped).

quota_recalc

Role repair. Counts the usage from the index again and stores it. Offline (the default) it locks the account and waits jobs.settle_seconds first, so that no delivery holds a reservation while the counters are replaced. With -online (or quota.online_recount auto) and the cluster mark quota/online (see shardpostctl-account(1)) it counts beside the writers, which capture every change of the usage while it runs, and replaces the counters in one compare-and-swap: no 451, no refused login, no BYE. The plan shows a dry count (approximate while the account is written to); the result reports the counted and the stored usage, the mode and, online, the barrier time, the captures and the switch rounds. A run that cannot start or switch safely (an open reservation older than the barrier, an import or a repairing scrub on the account, a switch that did not go through on an account with dozens of registry writes a second) ends busy and is planned again with a backoff; one that another writer invalidated (an offline recount, an import, a rename across the quota rules) ends lost and is planned again as well. Neither changes the counters. See docs/quota.md.

acl_reconcile

Role repair. Writes missing share pointers at the grantees and removes pointers that no longer carry rights.

fts_rebuild

Role repair, locks the account. Drops the full-text index (with its stored headers) and builds it again from every message. When it completes it sets the index version marker; without it, and while a rebuild runs or after one stopped midway, IMAP substring search does not use the index and checks every message of the mailbox. Run it for every account after an upgrade that changes how text is indexed. A layout 3 index gets the identifier filter (fts.dict_filter) and the text preparation (fts.attached_headers) of the configuration while the cluster mark of the dictionary filter is set (since 0.50.0): it is how an existing account changes its text preparation. It has no checkpoint: after a takeover it starts over.

With -online (since 0.51.0; needs the cluster mark of online runs, see shardpostctl-fts(1), else exit 65, and a layout 3 index, else exit 64) it does not lock the account and keeps the index: the target generation is indexed from the messages beside the current one, with the identifier filter and the text preparation of the configuration, the document ids stay, deliveries during the run are recorded and caught up, and one compare-and-swap switches. It sets the version marker of the target and removes the rebuild key, so it also heals an index an offline rebuild left midway. -skip-unreadable skips a live message whose blob cannot be read (counted as unreadable) instead of failing after five attempts; a rebuild that skipped one does not set the version marker nor remove the rebuild key (searches keep reading the messages instead of trusting an index that misses a mail); -keep-orphan-ids only counts the ids that have neither a blob nor a mailbox entry (orphan_ids) instead of dropping them. The result is the one of an online fts_layout with mode reindex. Without -online or -offline the plan decides: online with the mark and fts.online_build: auto, otherwise offline; the plan records the decision.

fts_dict_convert

Role repair. Writes the substring dictionary of the full-text index again from the posting lists and removes the objects of the layout used before 0.35.0. It reads no message and does not lock the account; it stops when an import or another job locks the account, and is then run again. The plan shows how many objects of the old layout are left; the result counts the terms by field, the dictionary entries written and the old objects removed. Running it again is harmless, and it is also the repair for a term that lost its dictionary entries (it is then found as a whole word but not as a part of one). It has no checkpoint.

A legacy kind since 0.46.0: it matters only for layout 2 indexes written before 0.35.0. A layout 3 index always has its dictionary (see fts_layout), so planning it for a layout 3 account is refused (exit status 65), and a run that finds layout 3 or a layout conversion under way, at its start or before any later batch, stops writing (no further batch, no deletion of old objects), succeeds and reports skipped in its result.

fts_layout

Role repair, locks the account. Converts the full-text index between layout 2 and layout 3, or reshards a layout 3 index (see shardpostctl-fts(1)). -layout 2 or 3 is the target (default the account's layout); -shards forces the object counts of a layout 3 target: one count for the posting, dictionary and header objects alike, or s/d/h for each (the parameters shards and shape, which exclude each other). Forced counts ignore fts.shards_max and fts.dict_shards_max (at most 4096). Otherwise the counts follow the key counts: posting and header objects from fts.shard_target_keys up to fts.shards_max, dictionary objects from fts.dict_target_keys up to fts.dict_shards_max (a conversion from layout 2 estimates them from the number of messages; the plan names the kinds a cap cut in its target, capped); -dict-filter sets the identifier filter of a layout 3 target (since 0.50.0, the parameter dict_filter): ids gives a body word with letters and digits of at least fts.dict_id_min_runes characters one start entry instead of a key per trigram, none takes the filter away; without it the target gets the filter that fts.dict_filter and the cluster mark of the filter ask for. The postings are copied as they are and the dictionary is derived from them by the target's filter, so both directions are exact; the dictionary objects of the target are estimated from a sample of the posting keys. The target keeps the source's text preparation (fts.attached_headers; changing it takes fts_rebuild). -due-only ends without a change while the key counts ask for the shape the index has and its identifier filter is the one asked for (the fts_due worker plans the job so, but never because of the filter). Converting to layout 3 needs the cluster mark (shardpostctl fts layout3 enable), -dict-filter ids the mark of the filter (shardpostctl fts dictfilter enable), read when the job is planned and when it starts. The job freezes the account's posting objects, writes the target generation from them (dead document ids left out, the dictionary derived from the terms, the header texts copied), switches the layout record and deletes the sources. A version before 0.49.0 knows a single cap, fts.shards_max, for all three kinds and plans a dictionary above it as due: on the way back (and while such a server holds the worker of fts.auto_reshard during a rolling update) set fts.auto_reshard: false or fts.shards_max to at least the largest D (shardpostctl fts layout3 list -verify), or its jobs shrink the dictionary. Deliveries into the account fail temporarily for the whole job (the account lock refuses them; one already running waits up to fts.frozen_wait_seconds for the switch). A job that fails or is cancelled removes its unfinished conversion itself; one that lost its lease leaves it to the server that takes over, which replaces it. The result names the source and target generations and counts terms, dictionary keys, header texts and documents, and since 0.49.0 the writes of the copy (writes, each fenced) and the most keys its write buffer held (buffer_keys); a noop result changed nothing. In a batch (plan -all) it runs per account with the batch's lock.

With -online (since 0.51.0; needs the cluster mark of online runs, else exit 65, and a layout 3 index, else exit 64; not with -layout 2) it does not lock the account: writers keep writing into the current generation and record what they index meanwhile, the copy builds the target beside it, the run catches up and switches in one compare-and-swap. Users notice at most a search that starts over once; an account with a steady stream of deliveries may hold its new deliveries up to fts.build_hold_ms (2 s) before the switch. The result adds online, mode (copy), rounds, caught_up, hold_ms, dead_ids and ends without a change and without an error as deferred busy (another run of the account alive, or the account locked), window or window_too_small (a run the fts_due worker or a batch planned that does not fit into the reshard window), mark_off (the cluster mark of online runs was removed after the plan), or as unsupported (no layout 3 index). Without -online or -offline the plan decides: online with the mark and fts.online_build: auto, otherwise offline; the plan records the decision, and a server before 0.51.0 that takes an online job over runs it offline under the lock.

mailbox_name_convert

Role repair, locks the account. Renames mailboxes that IMAP clients without UTF-8 support created under ShardPost 0.35.2 or earlier, when their modified UTF-7 names were stored as sent (for example Entw&APw-rfe for Entwürfe), to their UTF-8 names; children move with their parent. Subscriptions without a mailbox are rewritten the same way. Only a name level that is the exact modified UTF-7 form of another name changes; R&D stays. A rename whose new name exists already is reported as a conflict, with its children, and nothing is renamed for it; a name that is not valid UTF-8 is reported. The plan lists every change; check it before applying, since a mailbox that a UTF-8 client really named Entw&APw-rfe would be renamed as well. The usual conflict is a mailbox that delivery, Sieve or a JMAP client created in UTF-8 next to the one an older IMAP client created in modified UTF-7: merge the two by hand (move the messages, delete the empty one with mailbox_delete) and plan again. Sieve scripts, configured rules and client settings that name the old form are not rewritten; a fileinto of the old name creates that mailbox again, so change them to the UTF-8 names.

attachment_backfill

Role repair. Reads every message whose index record does not say whether it has an attachment — messages stored before the record had the field — and remembers the answer for JMAP (the hasAttachment filter of Email/query and the property of Email/get). It changes no message and no index record and does not lock the account; it stops when an import or another job locks the account. The plan shows how many such messages there are and how many are already remembered; the result counts the messages read, the ones without data, the ones that could not be read and the mailboxes that could not be listed, with the first errors; neither a broken message nor a broken mailbox ends the run. What a stopped run wrote stays, and running it again reads only what is left. A JMAP query starts the same fill on its own when it needs it; the job does it ahead of time.

mailbox_delete

Role delete. Deletes the mailbox named by -mailbox with its children. INBOX is refused. Offline (the default) it locks the account. With -online (or jobs.mailbox_delete_online auto) and the cluster mark mailbox/lock (see shardpostctl-mailbox(1)) it locks only the mailboxes it removes: a write into one is refused (IMAP NO [UNAVAILABLE], JMAP forbidden), a Sieve fileinto into one goes to INBOX, and only a session that has one selected is closed with BYE; everybody else notices nothing. The job locks the targets, waits jobs.settle_seconds and, at most jobs.mailbox_lock_barrier_seconds, for the operations that started before the lock, checks the plan (a delivery into a target since the plan is drift and the mail stays; flag changes are no drift online), unregisters the targets by id and frees the account before it releases blobs and shards. A target another job locks, an import or a repairing scrub on the account, or a MOVE into or out of a target that still stands (also an abandoned one, which the scrub completes once it is older than scrub.grace_seconds; shardpostctl plan scrub account and shardpostctl apply plan run that scrub now) ends the run busy, and it is planned again.

expunge

Role delete. Removes the messages named by -uids or, with -deleted, every message that carried \Deleted when the plan was made. Only messages that still have the UID and GUID from the plan are removed, at most 40000 per plan.

purge

Role delete, locks the account. Withdraws shares in both directions, deletes the learning jobs and every object of the account, removes it from the account directory and finally deletes its JMAP push subscriptions and submissions from the admin store. Until the directory step nothing can create the account again. Which objects go is purge.listing in the servers' configuration: true (the default) lists the account's namespaces in both pools and deletes what it finds, and reports in the log what the account's data model does not name; false deletes what the data model names (mailboxes with their shards, blobs, intents, the full-text index, METADATA, Sieve, the marks of public folders) and lists nothing; debris of servers before 0.42.0 then stays for shardpost-scrub(1) --sweep. The identity itself (LDAP) is not touched: if the account logs in again afterwards, it gets an empty INBOX. A purge that fails or is cancelled after it deleted the data leaves the account purging in the directory; every later job that locks it, a second purge included, then fails at once with "run shardpostctl account repair" (see shardpostctl-account(1)).

The account lock

Jobs marked above as "locks the account" take the same lock an import uses (online runs of fts_layout and fts_rebuild do not). While it is held, IMAP and POP3 answer "Mailbox maintenance in progress", LMTP answers 451 4.2.1, and other jobs and imports stay out. Sessions that were already open notice within five seconds and are disconnected. The job waits jobs.settle_seconds (10 by default) after locking before it writes, so deliveries in flight can finish.

If the account is already locked when a job starts, the job waits in state running with the phase "waiting for the account lock" and tries again every lease period, up to jobs.plan_ttl_seconds after it was created.

States

queued, running, succeeded, partially_failed, failed, cancel_requested, cancelled. job cancel stops a queued job at once and a running one at its next safe point; what it already did is not rolled back.