shardpost-system(1) – ShardPost docs

SHARDPOST-SYSTEM(1)

ShardPost Manual

SHARDPOST-SYSTEM(1)

NAME

shardpost-system - place the store-wide state in a system pool, check it, take it back

SYNOPSIS

shardpost [--config file] system status [--deep]
shardpost [--config file] system init --pool pool [--confirm run] [--wait]
shardpost [--config file] system migrate --to pool [--check] [--parallel n] [--force-registrations] [--confirm run] [--wait]
shardpost [--config file] system verify [--source-only [--delete-surplus --confirm run]] [--parallel n] [--wait]
shardpost [--config file] system abort --run run [--delete-copies] [--delete-surplus] [--confirm run] [--parallel n] [--wait]
shardpost [--config file] system revert --run run --confirm run [--delete-surplus] [--parallel n] [--wait]
shardpost [--config file] system cleanup --run run --confirm run [--min-age duration] [--delete-surplus] [--parallel n] [--wait]

DESCRIPTION

The store's own namespaces (~admin: admin store, jobs, registrations, auth flush log, queues; ~accounts: account directory and its queues; ~learning: the indexes of the learning queue; ~server: server metadata) live in the index pool, or, with ceph.system_pool, in a replicated pool of their own. These commands of the server binary put them there and take them back. They talk to the pools directly with the server's configuration and Ceph credentials, need no running server and no admin socket, and run in a Kubernetes Job of their own (hack/system-job.yaml). They never register in the admin store. Each prints a report as JSON on standard output; the log goes to standard error.

Which pool holds the state is decided by two objects, not by the configuration: the fence ~admin/system.moved in the index pool and the mark ~admin/system.pool in the system pool, both naming the pools by id and the cluster by fsid. A server whose ceph.system_pool does not match them refuses to start, and one that runs ends within a minute once they change (see docs/architecture/storage.md, "System pool and placement").

The commands that write hold the tool lease in the fence for as long as they work: a second tool is refused at once, naming the holder. The lease is renewed every 30 seconds; a tool that could not renew it for 90 seconds stops every write at once and ends with exit code 1 (running it again continues). A lease whose holder died is taken over with --wait once its holder was seen idle for three minutes on the taker's own clock; use that only when the Job or pod of the old holder is known to be gone, since a tool that was stopped (SIGSTOP, a frozen container) can still send operations it prepared once it wakes up. Never pause the Job of a tool. A lease whose last renewal lies more than 30 seconds ahead of the taker's clock is refused: run every host with NTP. The tools run with an OSD operation timeout of at most 60 seconds and refuse a ceph.client_options that sets more.

Every write command is resumable: run it again after an error, a crash or a deleted Job, and it continues where it stopped. Destructive steps need the run they belong to named with --confirm.

status

Shows the fence, the mark, the run state of a migration and the registrations of the processes in the admin store (in the system pool once the fence says so). With --deep it also counts the objects of the store's namespaces in both pools and of every quarantine (it lists the pools). It only reads.

init

Initializes the system pool of a new cluster: pool must be replicated and empty, and the index pool must hold nothing of the store's own namespaces (else the cluster exists: migrate it). The init creates the fence, writes the completion markers of the backfills into the empty indexes (so that the admin store and the learning queue read without a listing from the first start), writes the mark (the commit) and sets the fence to native. An init that ran before ends at once without writing. With --confirm run it deletes the content of an earlier run in the system pool (a retired mark and its copies, the run state of an aborted migration, the heads of an aborted init) and that run's quarantine. Then set ceph.system_pool on the server, the push worker and the scrub CronJob.

migrate

Moves the store's own namespaces of an existing cluster into the system pool pool, in a maintenance window with every server, push worker, scrub, backfill and sweep stopped. The phases, recorded in the run state ~admin/system.migrate of the system pool:

M0: the preconditions, after the lease was taken: the target is a pool of its own, keeps omap (a probe) and holds nothing but this run's work (or, confirmed, an earlier run's copies, which are deleted with that run's quarantine after the fence is set); no registration of a process that runs (seen within three minutes and not stopped), no valid lease of the push worker or of a running job, no held lock, no watcher on the objects every server watches (auth-flush, ix.jobs/claimable/00, ix.learning/00, ix.submission/00; each notified with a 2 second timeout).
M1: the fence becomes migrating (servers refuse from here on), the run state is created.
M2: one listing of the index pool: the source.
M3: the copy, in chunks of 4 MiB, with the xattrs (not the lock state) and the omap, and a SHA-256 digest per object (not repeated once M4 is done; a copy that stopped half way is replaced).
M4: the target holds exactly the source, both match the digests.
M5: the old objects move into the quarantine ns.moved/run of the index pool, each copy checked before its original goes; every old index head is saved in the run state and poisoned (format 1000), so that a server before 0.47.0 fails on it instead of reading an empty place.
M6: the mark: the commit.
M7: the fence becomes done; the report goes into the run state.

A migration that ended is a no-op that prints its report again. --check runs M0 without the lease and writes nothing but the probe: registrations, leases, watchers and held locks (the worker locks of the running servers) are reported with blocking false, not refused (the check before the processes are stopped); a target that is not empty, a failed probe or another wrong precondition still refuses. The migration itself refuses all of them.

verify

Without --source-only: compares the source of the migration (the old place, the quarantine, the saved heads) with the copy in the system pool, object by object. Run it before the servers use the system pool; once they write there, use --source-only. With --source-only: checks the old place after a migration: the store's namespaces of the index pool may hold the fence, the poisoned heads (unchanged, by the digest of M5) and anything under ~accounts/sweep/ (the sweep stays in the index pool), nothing else; the quarantine must have the digest of M4 until cleanup deleted it. Anything else is a surplus object, written by a process that does not know the system pool; find out which one, then delete them with --delete-surplus --confirm run (under the lease).

abort

Takes a migration (or an init) back before the servers use the system pool. The first call records the phase it found; a later call continues by it. Before M4: only the fence becomes reverted. From M4 on: the system pool must still have the digest of M4 (else a server used it: use revert); the quarantine is copied back to the original names, the heads get their saved values, surplus objects at the old place refuse (or are deleted with --delete-surplus and --confirm), the old place is checked against the digest of M4, the mark is retired and the fence becomes reverted. The quarantine is kept: delete it with cleanup after the minimum age, or with the next run's --confirm. An abort of a run that is aborted already checks the old place against the digest of M4 again and repairs it from the quarantine (a late write of a tool that was stopped); cleanup refuses to delete the quarantine while the old place differs. --delete-copies with --confirm also deletes the copies in the system pool, after the fence is reverted (run it again with the same options if it stopped there); the run state and the mark stay as the record of the run. Else the next run deletes the copies when its --confirm names this run.

revert

Takes a committed migration or init back, in a maintenance window: the system pool is the truth. With every process stopped (the checks of M0, on the system pool), the fence becomes migrating in the direction revert, every object of the system pool (but the placement objects) is copied to its name in the index pool, overwriting what differs, the poisoned heads included; objects of the index pool that the system pool does not have refuse until --delete-surplus deletes them; the index pool is checked against a fresh digest of the system pool, the quarantine is deleted, the mark retired and the fence set to reverted. Then clear ceph.system_pool everywhere. The copies stay in the system pool, retired, until a new init or migration deletes them.

cleanup

Deletes the quarantine of a migration once its fence is done, its mark active, the run at least --min-age old and the old place checks out as with verify --source-only. The fence and the poisoned heads stay for good; the fence records cleaned_at, and from then on verify --source-only expects no quarantine. It also takes an aborted run (fence reverted) and deletes only its quarantine.

OPTIONS

--config file: The server configuration (default: the environment variable SHARDPOST_CONFIG). ceph.index_pool, ceph.data_pool, ceph.user and the Rook secret apply as for a server. ceph.system_pool should be empty (the tools take the system pool from --pool, --to or the fence); a value that names another pool than the command is refused.
--pool pool: (init) The system pool to initialize.
--to pool: (migrate) The system pool to migrate into.
--run run: (abort, revert, cleanup) The run to work on, as status shows it (the fence's run).
--confirm run: Confirms a destructive step by naming the run it belongs to: deleting an earlier run's copies and quarantine (init, migrate), surplus objects or copies (verify, abort), a revert or a cleanup (required for both).
--check: (migrate) Only the preconditions, reporting running processes (registrations, leases, watchers, held locks) instead of refusing them.
--parallel n: Objects copied or hashed at a time (16; 1 to 64, other values are refused; not for status and init).Thetoolraisesthe objecter throttles of its connection to let them start (objecter_inflight_ops at least 4 per worker, objecter_inflight_op_bytes at least 4 MiB per worker) unless ceph.client_options sets them; values set there that are too small are refused.
--force-registrations: (migrate) Ignore registrations of processes that seem to run. Only for processes known to be dead (a pod killed without its orderly end); the report and the log (event=system_force_registration) name every registration ignored.
--source-only: (verify) Check the old place after the migration.
--delete-surplus: (verify with --source-only, abort, revert, cleanup) Delete the surplus objects at the old place (with --confirm).
--delete-copies: (abort) Delete the copies in the system pool (with --confirm).
--min-age duration: (cleanup) How long ago the run must have ended (168h).
--deep: (status) Count the objects of both pools.
--wait: Take over the lease of another tool once it was seen idle for three minutes. Only when that tool's Job or pod is known to be gone.

ROLES

None: the commands use the Ceph credentials of the server (client.shardpost in the manifests), not the admin API. Its caps must include the system pool.

EXIT STATUS

0: Done (also: nothing to do).
1: An error, a lost lease or an interruption; run the command again to continue.
2: Usage, or a precondition that does not hold: a running process, a lease held by another tool, a target that is not empty, surplus objects without --delete-surplus, a missing --confirm. The report names it; nothing was changed that the next run cannot take up. Right after the processes were stopped this is normal: the push worker's lease is valid for up to 90 seconds, job leases for up to 60, registrations without an orderly end for up to three minutes. Run it again after 90 seconds.
3: A check found the pools different from what they must be: a digest that does not match, an object that must not be there. Nothing is overwritten that must not be.

EXAMPLES

The check in normal operation, before the maintenance window:

shardpost system migrate --to shardpost-system --check

The migration, with every process stopped; the run id is in the report (run):

shardpost system migrate --to shardpost-system
shardpost system verify

After the servers run with ceph.system_pool:

shardpost system verify --source-only
shardpost system status

Seven days later, the quarantine:

shardpost system cleanup --run 4f2a9c1e07b3 --confirm 4f2a9c1e07b3

A migration that failed its checks, taken back before the servers started:

shardpost system abort --run 4f2a9c1e07b3

Back to the index pool after the switch, with the processes stopped:

shardpost system revert --run 4f2a9c1e07b3 --confirm 4f2a9c1e07b3

A new cluster:

shardpost system init --pool shardpost-system