2022-06-22T12:31:16.406 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T12:31:16.410 INFO:teuthology.results:subset: '7/16' 2022-06-22T12:31:16.411 INFO:teuthology.results:seed: '2962' 2022-06-22T12:36:08.754 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T12:36:08.759 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-22T13:38:14.383 INFO:teuthology.results:Tests finished! gathering results... 2022-06-22T13:38:16.691 INFO:scrape:Found 12 jobs 2022-06-22T13:38:19.899 INFO:scrape:Found 1 distinct failure reasons 2022-06-22T13:38:19.900 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-06-22T13:38:19.901 INFO:scrape:4 jobs: ['6892150', '6892152', '6892144', '6892142'] 2022-06-22T13:38:19.902 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/db/postgres}}', 'wsync/yes}'] 2022-06-22T13:38:19.903 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/db/postgres}}', 'wsync/yes}'] 2022-06-22T13:38:19.904 INFO:scrape: 2022-06-22T13:38:19.907 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [6892142] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:50:19 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892142/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892142/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892144] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:49:00 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892144/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892144/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892150] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:48:18 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892150/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892150/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6892152] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:52:15 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892152/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892152/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Pass ================================================================= [6892143] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/db/postgres}} time: 00:36:08 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892143/ [6892145] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/db/postgres}} time: 00:44:35 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892145/ [6892146] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/db/postgres}} time: 00:29:29 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892146/ [6892147] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/db/postgres}} time: 00:41:53 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892147/ [6892148] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/db/postgres}} time: 00:45:15 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892148/ [6892149] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/db/postgres}} time: 00:42:37 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892149/ [6892151] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/db/postgres}} time: 00:45:49 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892151/ [6892153] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/db/postgres}} time: 00:35:21 info: http://pulpito.ceph.com/pdonnell-2022-06-22_12:30:34-fs:workload-main-distro-default-smithi/6892153/