shardpost-scrub(1)

scrub

Scrubs accounts in this process and exits: the accounts of scrub.users, else every account of the account directory (once its backfill has completed), else every namespace of a listing of the index pool. The scrub checks and repairs what a crash can leave behind: reference counts, index entries without blob, unfinished MOVEs and deliveries, quota counters, and objects nothing points at. Where it looks for the last is scrub.orphan_source: listing lists each account's namespace in both pools, shadow (the default) does that and also works out what the intents name and reports where the two differ, intents reads the intents only and lists nothing. An account with an online quota recount under way (its run key qr in the registry) and one whose mailboxes a mailbox_delete job locks (an mblock: record before its destroying phase) are left alone (OnlineSkipped in the report, since 0.52.0). Before its first repair of entries or counters a scrub sets a repair mark (sr:<holder>) in the registry, which keeps a recount from starting beside it, and renews it with its lease; when a renewal fails the repairs stop (RepairsStopped). Captures a dead recount left behind are the finding quota_capture_leftover and are removed by the quota class. intents is safe only when every server of the cluster runs 0.42.0 or later: older servers write no removal intents, and a message they move or expunge can then look like an orphan to the scrub. Nothing checks this for you; switch only after the last older server is gone. With scrub.dry_run it only reports.

The scrub also looks at the layout of the account's full-text index (see shardpostctl-fts(1)). It reports, without repairing them, an object of the current layout 3 generation that is missing or carries another generation (FTSShardsMissing; searches and deliveries of the account fail until fts_rebuild rebuilds the index) and an object of layout 2 in an account whose layout 2 objects should be gone (FTSMixedLayout, written by a server before 0.45.0 after a wrong way back; repair with fts_rebuild). It enters an account with layout 3 in the presence index ix.fts_layout when it is missing there and removes an entry older than the grace period whose account has no layout 3 any more (FTSPresenceFixed). It reports a version marker of the index that does not fit its generation (FTSVersionDrift, fts_version_drift, since 0.50.0): the marker 3 of a dictionary without the identifier filter, which a version before 0.50.0 leaves when it converts a filtered generation (back to layout 2, or a reshard that drops the filter from the layout record), is set to 2 unless the run only reports (the dictionary is whole); a marker 2 on a generation with the filter or the text preparation nosig, which a version before 0.50.0 writes when it gives such a generation its first document ids while it is empty, is set to the marker of the generation (3 or 4), so that such a version no longer trusts it; a marker 4 whose layout record lost its text preparation is only reported (fts_layout writes the record anew). It also keeps the props of the presence entry in step with the generations (counted in FTSPresenceFixed). It reports the records of online runs of the full-text index that the layout record no longer names (FTSBuildLeftover, fts_build_leftover, since 0.51.0): keys w:<guid> and w# of the id map left by a run that crashed or that a version before 0.51.0 cleared. They are harmless; with the repair class uncovered they are removed, each page in a compare-and-swap with the layout record that keeps the keys of the run it names (a run that starts meanwhile keeps its records). The scrub job of the job system (shardpost scrub --batch, shardpostctl plan scrub) also removes an unfinished full-text conversion whose job does not run any more or whose import reshard stopped sending its heartbeat.

scrub --batch

Plans the scrub of every account as one batch job with skip_idle and submits it; the servers' job runners carry it out. This is what the nightly CronJob runs. See shardpostctl-job(1).

scrub --sweep

Finds what no account's data model names: debris that servers before 0.42.0 left without an intent, the objects of accounts purged before the purge named its objects, full-text shards above a lowered server.fts_shards, layout 3 generations of the full-text index the layout record does not name (a creation that lost a race, a conversion whose run crashed after its record was removed). The data model names every generation of the record (current, the conversion in progress, sources not removed yet), which costs one read of the record per account. It is a tool for a one-off cleanup, not a routine: it lists both pools in every namespace, which costs a walk over every placement group of each pool.

Pass 1 lists the index pool, then the data pool, and spills the object names into spill objects in the namespace ~accounts of the index pool (scrub.sweep_spill_shards per pool). Pass 2 reads them one namespace at a time and compares the names with the account's data model. What the model does not name is a remnant. Without --repair the sweep only reports; with it, a remnant older than scrub.grace_seconds is deleted (a blob through its reference count, so that a copy taken meanwhile keeps it).

A namespace without registry is deleted only when the account directory says the account is being purged (purging, a purge that stopped after the registry went), or a purge recorded it (the index ix.purged in ~accounts) and the directory holds no entry for it; an active entry always wins. A repairing sweep removes, when it ends, the records of the namespaces it cleaned completely; registering the name again removes the record too. Any other one is reported and left alone: a lost registry (registry_missing), an account that only kept Sieve scripts before 0.42.0 or the leftovers of a purge whose directory entry is gone (unregistered). An account whose data model cannot be read in full (a registry record that does not parse, an unreadable header, a registered mailbox without its shard 0) and an account under an import lock are left alone. The private \Seen marks of a live public folder that were written before 0.42.0 are adopted (their registry key ps: is written) rather than deleted.

The sweep checkpoints in the cluster (sweep/state in ~accounts): run it again after it stopped and it goes on where it was; in pass 1 it seeks to the listing position it reached, and lists a pool again when the pool's pg_num changed meanwhile. A run that goes on works with the flags it is given now: the namespaces it evaluated before keep what an earlier run without --repair only reported (use --restart to evaluate everything again). A sweep that ended has removed its spill objects and state; the next one lists the pools anew. One sweep runs at a time (a lock on sweep/lock in ~accounts). Lock, state and spill objects stay in the index pool when the rest of ~accounts lives in a system pool (ceph.system_pool); the account directory is read where it is. The store's own namespaces and every other reserved ~ namespace are never swept.

The report goes to standard output as one line of JSON after the log lines: the run, the objects listed per pool, the namespaces looked at, the remnants by class (blob, mailbox, marks, other, namespace for objects of a namespace without registry, fts for kept full-text shards above the count), the namespaces whose account was created again while the sweep looked at them (recreated; nothing of them is deleted), how many were removed, how many were left for being younger than the grace period, the marks adopted, a few samples and the first errors.

Run the sweep with the servers' configuration (the same server.fts_shards and pools). A shard of the full-text index (fts/, ftsh/, ftsd/) above the configured count in an account is reported as fts and never deleted: it may be a live one of servers configured with more. Such shards are left by lowering server.fts_shards; to remove them on purpose, rebuild the account's index with fts_rebuild (see shardpostctl-job(1)) and delete the shards above the new count by hand (rados rm). Any other remnant is deleted only at the object version that was judged, so a write in between leaves it for the next run.

The sweep needs the cluster permission mon 'allow r' (it asks the monitors for each pool's pg_num); the server's CephClient has it.