2022-09-11T08:23:03.870 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-11T08:23:03.872 INFO:teuthology.results:subset: '12/32' 2022-09-11T08:23:03.872 INFO:teuthology.results:seed: '599' 2022-09-11T10:18:08.219 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-11T10:18:08.243 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-11T20:53:52.603 INFO:teuthology.results:Tests finished! gathering results... 2022-09-11T20:53:56.476 INFO:scrape:Found 40 jobs 2022-09-11T20:54:04.397 INFO:scrape:Found 4 distinct failure reasons 2022-09-11T20:54:04.398 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi002 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' 2022-09-11T20:54:04.399 INFO:scrape:2 jobs: ['7024217', '7024233'] 2022-09-11T20:54:04.399 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2022-09-11T20:54:04.400 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2022-09-11T20:54:04.400 INFO:scrape: 2022-09-11T20:54:04.401 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-09-11T20:54:04.401 INFO:scrape:13 jobs: ['7024212', '7024201', '7024214', '7024222', '7024215', '7024206', '7024238', '7024199', '7024220', '7024231', '7024236', '7024209', '7024207'] 2022-09-11T20:54:04.402 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-09-11T20:54:04.403 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-09-11T20:54:04.403 INFO:scrape: 2022-09-11T20:54:04.404 INFO:scrape:Dead: b'2022-09-11T20:46:46.773 DEBUG:teuthology.exit:Finished running handlers' 2022-09-11T20:54:04.404 INFO:scrape:1 jobs: ['7024208'] 2022-09-11T20:54:04.404 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2022-09-11T20:54:04.405 INFO:scrape: 2022-09-11T20:54:04.405 INFO:scrape:Failure: Command failed on smithi055 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2d5edc8-31ae-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_4' 2022-09-11T20:54:04.406 INFO:scrape:2 jobs: ['7024202', '7024211'] 2022-09-11T20:54:04.406 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-09-11T20:54:04.407 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-09-11T20:54:04.407 INFO:scrape: 2022-09-11T20:54:04.410 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/ failed: 17 dead: 1 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [7024199] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:12:02 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024199/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024199/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024201] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:53:49 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024201/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024201/ error during scrub thrashing: rank damage found [7024202] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:14:33 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024202/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08a7829c43ae4e1b8d63db202e2b5704 Command failed on smithi055 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2d5edc8-31ae-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_4' [7024206] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:49:31 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024206/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024206/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024207] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:17:26 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024207/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024207/ error during scrub thrashing: rank damage found [7024209] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:59:30 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024209/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024209/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024211] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:12:20 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024211/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024211/ Command failed on smithi070 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid aa7f62b0-31b1-11ed-8431-001a4aab830c -- ceph daemon mds.e perf dump' [7024212] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:17:06 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024212/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024212/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024214] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:47:16 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024214/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024214/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024215] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:55:04 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024215/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024215/ error during scrub thrashing: rank damage found [7024217] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:54:00 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024217/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024217/ error during scrub thrashing: Command failed on smithi002 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7024220] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:18:32 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024220/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024220/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024222] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:48:58 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024222/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024222/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024231] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:40:31 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024231/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024231/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024233] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:50:49 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024233/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024233/ error during scrub thrashing: Command failed on smithi006 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7024236] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:18:04 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024236/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024236/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7024238] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:49:24 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024238/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024238/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Dead ================================================================= [7024208] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024208/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024208/ hit max job timeout Pass ================================================================= [7024200] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:17:02 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024200/ [7024203] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:07:52 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024203/ [7024204] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:19:03 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024204/ [7024205] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:53:50 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024205/ [7024210] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:48:06 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024210/ [7024213] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:51:55 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024213/ [7024216] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:16:32 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024216/ [7024218] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:46:45 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024218/ [7024219] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:20:49 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024219/ [7024221] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:51:45 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024221/ [7024223] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:12:31 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024223/ [7024224] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:17:38 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024224/ [7024225] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:51:24 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024225/ [7024226] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:44:59 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024226/ [7024227] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:11:20 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024227/ [7024228] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:18:37 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024228/ [7024229] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:54:03 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024229/ [7024230] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:46:23 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024230/ [7024232] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:19:02 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024232/ [7024234] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:40:37 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024234/ [7024235] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:09:52 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024235/ [7024237] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:48:19 info: http://pulpito.ceph.com/pdonnell-2022-09-10_13:11:17-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7024237/