2022-09-08T17:42:13.884 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-08T17:42:13.887 INFO:teuthology.results:subset: '6/64' 2022-09-08T17:42:13.887 INFO:teuthology.results:seed: '5578' 2022-09-08T17:54:11.171 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-08T17:54:11.199 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-08T19:55:20.795 INFO:teuthology.results:Tests finished! gathering results... 2022-09-08T19:55:21.905 INFO:scrape:Found 10 jobs 2022-09-08T19:55:23.668 INFO:scrape:Found 2 distinct failure reasons 2022-09-08T19:55:23.669 INFO:scrape:Failure: error during scrub thrashing: rank damage found 2022-09-08T19:55:23.670 INFO:scrape:3 jobs: ['7020940', '7020946', '7020948'] 2022-09-08T19:55:23.671 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-09-08T19:55:23.671 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-09-08T19:55:23.672 INFO:scrape: 2022-09-08T19:55:23.673 INFO:scrape:Failure: 2022-09-08T19:55:23.673 INFO:scrape:1 jobs: ['7020942'] 2022-09-08T19:55:23.674 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}'] 2022-09-08T19:55:23.675 INFO:scrape: 2022-09-08T19:55:23.676 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7020940] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:20:57 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020940/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020940/ error during scrub thrashing: rank damage found [7020942] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:39 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020942/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ee028b7ba7249b49eb1e0e937dfe476 [7020946] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:36:40 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020946/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020946/ error during scrub thrashing: rank damage found [7020948] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:37:46 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020948/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020948/ error during scrub thrashing: rank damage found Pass ================================================================= [7020939] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:28:29 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020939/ [7020941] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 01:00:28 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020941/ [7020943] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:14 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020943/ [7020944] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:58:47 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020944/ [7020945] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 01:12:04 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020945/ [7020947] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:30:33 info: http://pulpito.ceph.com/pdonnell-2022-09-08_17:41:42-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7020947/