2022-09-10T08:45:34.137 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-10T08:45:34.139 INFO:teuthology.results:subset: '13/32' 2022-09-10T08:45:34.139 INFO:teuthology.results:seed: '7334' 2022-09-10T09:50:16.930 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-10T09:50:16.952 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-10T12:40:30.714 INFO:teuthology.results:Tests finished! gathering results... 2022-09-10T12:40:34.884 INFO:scrape:Found 40 jobs 2022-09-10T12:40:45.899 INFO:scrape:Found 3 distinct failure reasons 2022-09-10T12:40:45.901 INFO:scrape:Failure: 2022-09-10T12:40:45.901 INFO:scrape:30 jobs 2022-09-10T12:40:45.902 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2022-09-10T12:40:45.903 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-09-10T12:40:45.903 INFO:scrape: 2022-09-10T12:40:45.904 INFO:scrape:Failure: Command failed on smithi077 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 37882db0-30e8-11ed-8431-001a4aab830c -- ceph daemon mds.h perf dump' 2022-09-10T12:40:45.904 INFO:scrape:1 jobs: ['7023795'] 2022-09-10T12:40:45.905 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}'] 2022-09-10T12:40:45.905 INFO:scrape: 2022-09-10T12:40:45.906 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-09-10T12:40:45.906 INFO:scrape:3 jobs: ['7023807', '7023791', '7023799'] 2022-09-10T12:40:45.907 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-09-10T12:40:45.907 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2022-09-10T12:40:45.908 INFO:scrape: 2022-09-10T12:40:45.910 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/ failed: 34 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7023791] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:02:37 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023791/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023791/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7023792] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:42 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023792/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42b39a0968714817823ee6be0f221b57 [7023793] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:29:25 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023793/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3869f115c43947069b18dee8f2cc7f2d [7023794] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:37 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023794/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dc00026839e4be8a49fe1ce78464b5c [7023795] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 03:41:38 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023795/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023795/ Command failed on smithi077 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 37882db0-30e8-11ed-8431-001a4aab830c -- ceph daemon mds.h perf dump' [7023796] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:25:24 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023796/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1644fe98ecc4451abfa56eb1b0f7c0ae [7023797] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:13 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023797/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f794676ddc348fbb6e43de7ad514abc [7023798] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:25 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023798/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7d8fe33527241e4bdd3139819a09ed0 [7023799] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:05:23 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023799/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023799/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7023800] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:38 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023800/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ed8523e01004e8798fc73e8e35ac980 [7023801] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:23 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023801/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c239844d30d4b508aa0efd9bf79cc5f [7023802] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:08 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023802/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023802/ sentry: https://sentry.ceph.com/organizations/ceph/?query=268220b368234fa8bb8cd7b472bf91e9 [7023804] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:52 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023804/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de2349834aa345c79dd4832ab95a1a70 [7023805] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:28 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023805/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a683bed2ac14265822f7ef526c412f3 [7023806] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:41 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023806/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95dd0fee2bcd488bb54fbc2239180630 [7023807] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:04:55 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023807/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023807/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7023808] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:40 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023808/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f765753d59d4923b3de7d0ad3d7bdee [7023809] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:40 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023809/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c612c7b8e7e64c1db90d8db04799d979 [7023810] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:25 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023810/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7538de56ec3e4c2caece4a2fee0de995 [7023812] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:56 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023812/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ea2c88e45714f5798b70b380eca331a [7023813] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:31:15 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023813/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023813/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5ff4e4c11d84c54ac3cd0b7ad973bcc [7023814] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:29 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023814/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d35df918c644e3896c6a3edf68e1a61 [7023816] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:28:48 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023816/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4bd5d34066aa4a7f87f44a1c63e9aa15 [7023817] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:28 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023817/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0483f71e4029453087ebd4e97425f2c3 [7023818] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:53 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023818/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98509705dd094642963436dab87f4d46 [7023820] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:34 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023820/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=101b7e35a0ed45b981ce1c852dc86424 [7023821] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:29:43 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023821/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=160ec11f26d24d2bbb37c462139d4875 [7023822] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:56 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023822/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77e41385939e40a0b954a9e1494b55c0 [7023824] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:45 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023824/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92b5d1d6da6345c39855d24f42ae2574 [7023825] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:41 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023825/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b5452a6e1dd42fca4810281cbcfa7ad [7023826] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:26:42 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023826/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7a57379845b42eaa3a1d55b6f9441e6 [7023828] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:53 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023828/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe04c31c8f744d1ea83e7778013040d3 [7023829] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:31 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023829/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f22ca3c4eec44ff59bf5326b755862a7 [7023830] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:03 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023830/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff306e0aa51748e590b296d58f982f4a Pass ================================================================= [7023803] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:10:20 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023803/ [7023811] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:07:06 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023811/ [7023815] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:14:35 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023815/ [7023819] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:09:41 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023819/ [7023823] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:09:36 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023823/ [7023827] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 02:11:28 info: http://pulpito.ceph.com/pdonnell-2022-09-09_18:55:40-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023827/