ShardPost

Manual pages

shardpost-bench(1)

Fill separate bench pools with accounts and measure what an account costs in Ceph.

SHARDPOST-BENCH(1) ShardPost Manual SHARDPOST-BENCH(1)

NAME

shardpost-bench - fill separate bench pools with accounts and measure what an account costs in Ceph

SYNOPSIS

shardpost [--config file] bench populate --index-pool pool --data-pool pool [--system-pool pool] [options]
shardpost [--config file] bench measure --index-pool pool --data-pool pool [--system-pool pool] [options]

DESCRIPTION

The bench is the measurement of the scaling plan's stage R4: objects and bytes per account, RADOS operations per delivery, delivery latency, compare-and-swap retries and the cost of opening an account. It is a command of the server binary, not of shardpostctl(1): it links the RADOS store and writes through it directly, with the server's configuration and Ceph credentials. It needs neither a running server nor Nauthilus, and it does not use IMAP or LMTP. It runs in a Kubernetes Job of its own with the server's image (hack/bench-job.yaml).

The bench writes only into two pools of its own (three with --system-pool), which the operator creates for it and deletes afterwards (docs/operations/bench.md). It refuses to run (exit status 2):

  • without --index-pool and --data-pool, when one of the bench pools names ceph.index_pool, ceph.data_pool or ceph.system_pool of the configuration (file or environment) or the built-in defaults shardpost-index and shardpost-data, or when two bench pools are the same pool;
  • with an account prefix that does not start with bench-;
  • measure on pools that carry no bench marker (the object marker in the namespace ~bench, written by the first populate);
  • populate on a pool without the marker that holds any object, in the index or system pool in particular an account directory (ix.accounts/head in ~accounts), in the index pool also a placement fence (system.moved in ~admin): a pool somebody uses. The marker is written, pool by pool, only into a pool that is empty. The objects rbd_info and rbd_trash that rbd pool init leaves in the default namespace do not count (Rook initialises every CephBlockPool for RBD);
  • when the placement check of the bench pools fails (below).

Everything the store keeps beside the accounts (the account directory in ~accounts, the admin store in ~admin) lands in the bench pools as well: in the bench system pool with --system-pool, otherwise in the bench index pool. The production system pool of ceph.system_pool is never taken over; ~admin and ~accounts of the production pools are never touched.

The bench opens the store as a tool and checks the placement once the pools are set up. With --system-pool, populate initializes the system pool after the markers with the routine of shardpost system init (the fence in the bench index pool, the completion markers of the empty indexes, the mark in the bench system pool); a second populate finds it initialized and writes nothing. The init refuses a bench index pool that already holds the store's own namespaces: a run with a system pool needs fresh bench pools. Then populate and measure check the placement as a server does: a bench index pool with a fence and no --system-pool is refused.

bench populate

Creates the accounts prefixNNNNNN (six digits, from --first on) with the mailboxes of --mailboxes and delivers messages until every account holds --messages. Each message goes through the delivery pipeline LMTP uses (the Sieve run, the delivery headers, the reservation, the blob, the full-text index, the index entry and the account change log); a share of --inbox-share goes to INBOX through Sieve, the rest evenly into the other mailboxes, named in the request as dovecot-lda -m does. The pipeline has no relay, no external programs and none of the global, before or after scripts of the configuration.

The messages are generated: the size is drawn from --sizes, the text from a Zipf distribution over a vocabulary of 50000 terms (so the full-text index sees frequent and rare terms), and a message above 32 KiB carries a base64 attachment for the rest of its size. Message n of an account is the same for the same --seed and --corpus.

--corpus ids adds what real mail carries and the plain vocabulary does not: identifiers new in every message. Each message has a reference of 32 hex digits (Ref: ), sometimes hex ids of 16, 32 or 64 digits, up to one tracking link with a base64url token of 20 to 120 characters, a line of 40 to 70 short numbers, codes and names, in 2 % of the messages a base64 block in the text, and in 5 % a forwarded message (message/rfc822) or a bounce (text/rfc822-headers) with full headers: three Received fields, ARC, authentication results, DKIM and X-Google-DKIM signatures. --corpus ids,nosig is the same messages from the same random draws without the signature and trace fields of those attached headers (DKIM-Signature, ARC-Seal, ARC-Message-Signature, ARC-Authentication-Results, Authentication-Results, Received, Received-SPF, X-Google-DKIM-Signature, X-*-Signature): the difference of two runs is what those fields cost the dictionary. One account of 12 000 messages of the default sizes has about 50 terms and 325 dictionary keys per message with ids, about 45 and 230 with ids,nosig, and about 10 and 40 with the plain corpus, whose vocabulary is exhausted by then (the largest real account measured: 57 and 280 at 36 000 messages). The same messages can be delivered to a real test account over LMTP with shardpost-load -corpus. An account continues at its current message count, so a run that was stopped is resumed by running it again with the same options, and a run with a higher --messages adds to the accounts.

The report gives the deliveries, their throughput, the delivery latency (p50, p90, p99, maximum; one delivery per account at a time, --parallel accounts at once) and the compare-and-swap retries per delivery by object kind.

bench measure

Takes the inventory of the bench accounts and counts operations. The inventory lists both bench pools in every namespace and reads the size and, in the index pool, the omap of every object of the accounts (every --sample-every-th account by number) and of the store's own namespaces. It reports per account the objects, bytes, omap keys and omap bytes by pool and object class: registry, account, mbox (header and first UID shard), mbox-shard, msg (data pool), fts (posting lists), ftst (trigram dictionary), ftsh (header store), ftsd (dictionary before 0.35.0), ftx-p, ftx-d, ftx-h (posting, dictionary and header objects of a layout 3 index, ftx/<gen>/p<i>, d<j>, h<k>), ftsmap, metadata, sieve, scrub, ix (front-end indexes), pubseen, store-ix (the store's own indexes) and other; the mean, minimum and maximum over the accounts, the usage per message, an estimate of the raw space the data objects take in an erasure coded pool, and the means scaled to --extrapolate accounts.

Then --probe-accounts accounts, spread over the inventory, get --probe-messages more messages each in two rounds: the first measures latency and compare-and-swap retries, the second, on a connection of its own with the client library's objecter log switched on, counts every operation sent to an OSD by pool, object class and kind (read, write, class method call). The same accounts are opened in two more pairs of rounds: Store.User() and Close() alone, and the opening of a session (Store.User(), the mailbox list, the state of INBOX). The report gives operations per delivery and per open, the latencies, and the Go heap one open view holds. The probe messages stay in the accounts; the next populate counts them.

With --contention p two more delivery rounds follow in the same way, in which p deliveries at a time go into one probe account (each of them --probe-messages long; the accounts one after the other). The deliveries then contend for the same objects of the account; the report gives their latency, the deliveries per second, the compare-and-swap retries by kind and by object class, and the operations per delivery, retries included.

With --search k the probe accounts first answer k search queries each, before any delivery, in two rounds like the deliveries. The queries are a fixed set drawn from the vocabulary of the message generator (the same for every account and every run): whole words, word starts and parts from inside a word of 3 to 6 characters, frequent and rare words, as TEXT, BODY, SUBJECT and FROM. With --corpus ids every second query is an identifier query instead, as BODY and TEXT: the reference of one of the first five messages of the account as a whole (id-whole), its first 8 characters (id-start) and 8 characters from inside it (id-inner); the report gives the latency per kind of query. Each query asks the full-text index the way IMAP SEARCH does (the candidates of a substring search, TEXT as header and body) and the way the word search does, in a view of its own. The report gives the latency and the operations per query, and per query the messages found and a SHA-256 fingerprint of the results (account, field set, GUID); the fingerprint of all queries compares the results of two runs. A run with --probe-messages 0 and without --expunge changes nothing, so the next run on the same accounts must report the same fingerprint (for instance before and after a conversion of the index).

With --expunge n the probe accounts lose their n oldest INBOX messages in each of two more rounds, at the end of the run, one EXPUNGE per message in a view of its own; the report gives the latency, the compare-and-swap retries and the operations per expunge.

With --convert (and layout 3: --fts-layout 3 or, without the flag, fts.layout 3 of the configuration, the default) the probe accounts' full-text index is converted to layout 3 after the search probe, as a fts_layout job would do it: account lock, conversion, unlock. The latency is that of the first round, which converts from layout 2: the time the account is locked, during which deliveries fail temporarily (the 451 window of the job). The operations are counted in the second round, which reshards the converted index once more. With --search the search probe runs again after the conversion and must report the same fingerprint; a note says when it does not.

With --online as well (since 0.51.0) the bench sets the mark of online runs in its pools and reshards every probe account once more online, one after the other, while --load r deliveries a second go into the account of the run. It reports the latency of the runs, their catch-up rounds, the documents caught up and the time new deliveries were held before the switches, and the count, latency and failures of the deliveries during the runs (failures should be 0: an online run refuses no delivery; a note names them otherwise).

The objecter log (debug_objecter 10, one line per operation) is written to --trace-dir and parsed when the round ends; it holds about 2.5 KB per operation. Lines of a format the parser does not know are not counted; the report says how many there were.

RocksDB and onode caches

The OSD's caches come from Ceph itself; the bench does not ask for them. While the bench pools are filled, on a host with the admin key (the Rook toolbox):

ceph tell osd.0 perf dump bluestore | jq '.bluestore | with_entries(select(.key | test("onode")))'
ceph tell osd.0 perf dump rocksdb
ceph tell osd.0 dump_mempools | jq '.mempool.by_pool | {bluestore_cache_onode, bluestore_cache_meta, bluestore_cache_data, bluestore_cache_other}'
ceph tell osd.0 config get osd_memory_target
ceph tell osd.0 bluestore allocator score block
ceph osd df tree
ceph df detail


Take them before populate, after it, and after
    measure, for every OSD that holds placement groups of the bench pools
    (ceph pg ls-by-pool shardpost-load-index).








OPTIONS


Both subcommands:



  
--config
    file

  
The server configuration (default: the environment variable
      SHARDPOST_CONFIG). Its ceph.index_pool,
      ceph.data_pool and ceph.system_pool are the production pools
      the guard compares with; the rest (ceph.user, the Rook secret,
      server.fts_shards, server.mbox_shard_size,
      ceph.chunk_size, server.quota_bytes, fts.*) applies
      to the bench as to a server.

  
--fts-layout
    n

  
The layout of full-text indexes the run creates: 2 or 3 (0, the default:
      fts.layout of the configuration, which is 3 unless the
      configuration sets it). With 3 the bench sets the layout 3 mark in the
      admin namespace of the bench pools (never in production pools: the guard
      checks them first).

  
--index-pool
    pool

  
The bench index pool (replicated; required).

  
--data-pool
    pool

  
The bench data pool (erasure coded like production; required).

  
--system-pool
    pool

  
The bench system pool (replicated; optional): the store's own namespaces
      of the run live there, as with ceph.system_pool. It must not be a
      production pool nor one of the other bench pools. Without it they live in
      the bench index pool, whatever the configuration says.

  
--prefix
    prefix

  
The account name prefix; it must start with bench- (default
      bench-).

  
--parallel
    n

  
Accounts worked on at the same time; for measure, the inventory's
      readers (4). Run it off-peak: the bench pools share the OSDs with
      production.

  
--sizes
    dist

  
The message size distribution,
      weight:min-max separated by commas,
      sizes in bytes or with KiB, MiB (default
      60:1KiB-8KiB,30:8KiB-64KiB,10:64KiB-1MiB).

  
--seed
    n

  
The seed of the message generator (1).

  
--corpus
    name

  
The content of the messages: plain (the default), ids (with
      identifiers and attached headers) or ids,nosig (the same without
      the signature and trace fields of the attached headers). Populate and
      measure an account with the same corpus.

  
--json
    file

  
Also write the report as JSON to file; - writes it to
      standard output and the table to standard error.




populate:



  
--accounts
    n

  
Number of accounts (1000).

  
--first
    n

  
Number of the first account (0); several Jobs can fill disjoint
    ranges.

  
--messages
    n

  
Messages every account ends up with (50).

  
--mailboxes
    list

  
The mailboxes of every account, INBOX first
      (INBOX,Sent,Drafts,Trash,Junk,Archive; the five after INBOX get
      their special use).

  
--inbox-share
    f

  
Share of the messages delivered to INBOX through Sieve (0.7).

  
--progress
    duration

  
Interval of the progress log (30s; 0 for none).




measure:



  
--sample-every
    n

  
Take the inventory of every n-th account by number (1: all).

  
--probe-accounts
    n

  
Accounts of the delivery and open rounds (20; 0 for the inventory
    only).

  
--probe-messages
    n

  
Messages per probe account and delivery round (10; 0 for no delivery
      rounds).

  
--contention
    p

  
Add two delivery rounds with p deliveries at a time into each probe
      account (0, the default: none). Not to be confused with --parallel,
      which works on several accounts at once.

  
  
Search queries per probe account and round (0, the default: no search
      rounds). With --probe-accounts 0 there are no probe accounts and no
      search.

  
--expunge
    n

  
INBOX messages to expunge per probe account and round (0, the default: no
      expunge rounds). The messages are gone afterwards. Like --search,
      it does nothing with --probe-accounts 0.

  
--convert

  
measure: convert the probe accounts to layout 3 and reshard them
      once more (needs layout 3: --fts-layout 3 or, without the flag,
      fts.layout 3, the default).

  
--online

  
measure --convert: reshard the probe accounts online once more
      (sets the mark of online runs in the bench pools).

  
--load
    r

  
measure --convert --online: deliveries per second into the account
      of each online reshard (0, the default: none).

  
--extrapolate
    n

  
Scale the per-account means to n accounts (1000000; 0 for
    none).

  
--ec-k k, --ec-m
    m

  
The erasure code profile of the data pool for the raw space estimate (2
      and 1).

  
--stripe-unit
    bytes

  
The profile's stripe unit (4096).

  
--min-alloc
    bytes

  
BlueStore's allocation unit on the data pool's OSDs (4096).

  
--trace-dir
    dir

  
Where the objecter logs go (the temporary directory, below
      shardpost-bench).

  
--keep-traces

  
Keep the objecter logs.








PERMISSIONS


The cephx user of the configuration needs osd 'allow rwx
    pool=<bench index pool>, allow rwx pool=<bench data pool>'
    and mon 'allow r'. and nothing on the production pools. No admin API
    role is involved. In Kubernetes the bench runs as
    client.shardpost-bench (the CephClient in
    hack/bench-pools.yaml, caps on the bench pools only) and reads its
    key the way the server does, from the Rook secret
    rook-ceph-client-shardpost-bench, through the ServiceAccount
    shardpost-bench.






OUTPUT


A table on standard output (standard error with --json -),
    logs on standard error, and with --json the report as JSON. The JSON
    of measure has the keys
    inventory.per_account.pool.class (spreads of
    objects, bytes, omap_keys, omap_bytes), inventory.per_message,
    store_wide, data_raw, extrapolated, pools,
    delivery (parallel, latency, deliveries_per_second, cas_retries
    by_kind and by_class, ops_per_delivery by pool and class), contention
    (the same, with --contention), search (count, latency,
    ops_per_call, queries with their key, kind, text, candidates, words and
    fingerprint, and the fingerprint of all; with --search),
    expunge (count, latency, cas_retries, ops_per_call; with
    --expunge), convert and search_after_convert (with
    --convert), convert_online (accounts, load, run_latency,
    catchup_rounds, caught_up, hold_ms, deliveries, failures, delivery_latency;
    with --online), user_open and session_open.






EXIT
  STATUS



  
0

  
The run ended and every account was filled (populate) or measured.

  
1

  
Something failed: the cluster, the credentials, the inventory, a probe
      round, or accounts that could not be filled (the report says how many; run
      populate again to finish them).

  
2

  
A usage error or a refusal of the guard: no or production pools, a prefix
      outside bench-, pools without the bench marker, a pool in use, or a
      placement of the bench pools that does not match --system-pool (or
      a system pool the init refuses).








EXAMPLES


Fill 1000 accounts with 50 messages each, then measure:






shardpost bench populate --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --accounts 1000 --messages 50
shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --json /tmp/measure.json






Grow the same accounts to 500 messages and measure a tenth of
    them:






shardpost bench populate --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --messages 500
shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --sample-every 10






Fresh bench pools with a system pool of their own, to count the
    operations that reach it:






shardpost bench populate --index-pool shardpost-load-index \


    --data-pool shardpost-load-data \


    --system-pool shardpost-load-system
shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data \


    --system-pool shardpost-load-system --json /tmp/system.json






Eight deliveries at a time into each of 20 accounts, ten messages
    each, to measure compare-and-swap contention:






shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --contention 8 \


    --json /tmp/contention.json
jq '.contention.cas_retries' /tmp/contention.json






Compare the search results of the same accounts before and after a
    change of the index, and measure what an expunge costs:






shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --probe-messages 0 \


    --search 24 --json /tmp/search-before.json
# ... change the index, then the same command with search-after.json
jq -r .search.fingerprint /tmp/search-before.json /tmp/search-after.json
shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --probe-messages 0 --expunge 5






New accounts in layout 3, and the conversion of layout 2 accounts
    with the search results compared before and after:






shardpost bench populate --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --prefix bench-l3 --fts-layout 3
shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --probe-accounts 100 \


    --probe-messages 0 --search 12 --fts-layout 3 --convert






The same, then online reshards with eight deliveries a second into
    the account of each run (no delivery may fail):






shardpost bench measure --index-pool shardpost-load-index \


    --data-pool shardpost-load-data --probe-accounts 20 \


    --probe-messages 0 --convert --online --load 8 \


    --json /tmp/online.json
jq '.convert_online' /tmp/online.json






The operations of a delivery, by object class, from the JSON:






jq '.delivery.ops_per_delivery' /tmp/measure.json










SEE
  ALSO


shardpost-scrub(1), shardpostctl(1)







  

    2026-10-03
    ShardPost 0.51.0
  




Rendered from docs/man/man1/shardpost-bench.1 in the ShardPost repository.