2022-06-22T16:55:12.790 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T16:55:12.792 INFO:teuthology.results:subset: '5/16' 2022-06-22T16:55:12.792 INFO:teuthology.results:seed: '4298' 2022-06-22T16:55:59.144 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T16:55:59.148 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-22T18:41:10.009 INFO:teuthology.results:Tests finished! gathering results... 2022-06-22T18:41:13.117 INFO:scrape:Found 6 jobs 2022-06-22T18:41:17.195 INFO:scrape:Found 1 distinct failure reasons 2022-06-22T18:41:17.206 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-06-22T18:41:17.207 INFO:scrape:5 jobs: ['6892260', '6892264', '6892262', '6892261', '6892265'] 2022-06-22T18:41:17.213 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'session_timeout}', 'standby-replay'] 2022-06-22T18:41:17.214 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2022-06-22T18:41:17.216 INFO:scrape: 2022-06-22T18:41:17.223 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6892260] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} ----------------------------------------------------------------- time: 01:38:26 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892260/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892260/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892261] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} ----------------------------------------------------------------- time: 01:13:39 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892261/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892261/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892262] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} ----------------------------------------------------------------- time: 01:32:36 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892262/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892262/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892264] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} ----------------------------------------------------------------- time: 01:18:22 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892264/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892264/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892265] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} ----------------------------------------------------------------- time: 01:00:46 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892265/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892265/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Pass ================================================================= [6892263] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-workunit/db/postgres}} time: 01:02:01 info: http://pulpito.ceph.com/pdonnell-2022-06-22_16:55:05-fs:workload-main-distro-default-smithi/6892263/