2022-05-18T07:36:17.785 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-18T07:36:17.788 INFO:teuthology.results:subset: '22/32' 2022-05-18T07:36:17.788 INFO:teuthology.results:seed: '939' 2022-05-18T08:25:18.677 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-18T08:25:18.700 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-18T14:38:48.002 INFO:teuthology.results:Tests finished! gathering results... 2022-05-18T14:38:55.621 INFO:scrape:Found 48 jobs 2022-05-18T14:39:10.251 INFO:scrape:Found 10 distinct failure reasons 2022-05-18T14:39:10.252 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2022-05-18T14:39:10.253 INFO:scrape:1 jobs: ['6839547'] 2022-05-18T14:39:10.253 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}', 'wsync/{yes}}'] 2022-05-18T14:39:10.254 INFO:scrape: 2022-05-18T14:39:10.254 INFO:scrape:Failure: "2022-05-18T09:25:10.052880+0000 mon.b (mon.1) 509 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log 2022-05-18T14:39:10.255 INFO:scrape:2 jobs: ['6839538', '6839544'] 2022-05-18T14:39:10.255 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.256 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{no_client_pidfile', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/strays}', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.256 INFO:scrape: 2022-05-18T14:39:10.257 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2022-05-18T14:39:10.257 INFO:scrape:1 jobs: ['6839534'] 2022-05-18T14:39:10.257 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/pool-perm}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.258 INFO:scrape: 2022-05-18T14:39:10.258 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-05-18T14:39:10.259 INFO:scrape:2 jobs: ['6839529', '6839515'] 2022-05-18T14:39:10.259 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.260 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/snaps}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}'] 2022-05-18T14:39:10.260 INFO:scrape: 2022-05-18T14:39:10.260 INFO:scrape:Failure: Test failure: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-05-18T14:39:10.261 INFO:scrape:2 jobs: ['6839508', '6839532'] 2022-05-18T14:39:10.261 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{whitelist_health', 'tasks/cephfs_misc_tests}', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.262 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{whitelist_health', 'tasks/cephfs_misc_tests}', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.262 INFO:scrape: 2022-05-18T14:39:10.263 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2022-05-18T14:39:10.263 INFO:scrape:1 jobs: ['6839554'] 2022-05-18T14:39:10.264 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.264 INFO:scrape: 2022-05-18T14:39:10.265 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 2022-05-18T14:39:10.265 INFO:scrape:1 jobs: ['6839523'] 2022-05-18T14:39:10.266 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.266 INFO:scrape: 2022-05-18T14:39:10.266 INFO:scrape:Dead: b'2022-05-18T14:34:50.769 DEBUG:teuthology.exit:Finished running handlers' 2022-05-18T14:39:10.267 INFO:scrape:1 jobs: ['6839535'] 2022-05-18T14:39:10.267 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/recovery-fs}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.268 INFO:scrape: 2022-05-18T14:39:10.268 INFO:scrape:Failure: Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2022-05-18T14:39:10.269 INFO:scrape:1 jobs: ['6839552'] 2022-05-18T14:39:10.269 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}', 'wsync/{no}}'] 2022-05-18T14:39:10.269 INFO:scrape: 2022-05-18T14:39:10.270 INFO:scrape:Failure: "2022-05-18T11:33:45.293550+0000 osd.2 (osd.2) 24 : cluster [ERR] 5.1bs0 deep-scrub : stat mismatch, got 256/257 objects, 0/1 clones, 256/257 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/1 whiteouts, 1787112/1787112 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2022-05-18T14:39:10.270 INFO:scrape:1 jobs: ['6839514'] 2022-05-18T14:39:10.271 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-18T14:39:10.271 INFO:scrape: 2022-05-18T14:39:10.274 INFO:teuthology.results:Sending results to nshelke@redhat.com: Test Run: nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/ failed: 12 dead: 1 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [6839508] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:24:14 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839508/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70080c610a264ca596cbb3e1235c742f Test failure: test_fs_new (tasks.cephfs.test_misc.TestMisc) [6839514] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:09:21 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839514/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839514/ "2022-05-18T11:33:45.293550+0000 osd.2 (osd.2) 24 : cluster [ERR] 5.1bs0 deep-scrub : stat mismatch, got 256/257 objects, 0/1 clones, 256/257 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/1 whiteouts, 1787112/1787112 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log [6839515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:21:57 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839515/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839515/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bca4033d62e1422399e3fab1b6b6a327 reached maximum tries (90) after waiting for 540 seconds [6839523] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} ----------------------------------------------------------------- time: 00:13:14 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839523/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a01dec66944c4fcb955a31708010b71b Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) [6839529] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 03:00:24 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839529/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839529/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6839532] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839532/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839532/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd215ef1d4f4fdfbde83868625550c9 Test failure: test_fs_new (tasks.cephfs.test_misc.TestMisc) [6839534] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} ----------------------------------------------------------------- time: 00:17:35 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839534/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839534/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1afe4509bcc94eb19f89762719ef0232 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [6839538] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:57:38 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839538/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839538/ "2022-05-18T09:25:10.052880+0000 mon.b (mon.1) 509 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log [6839544] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} ----------------------------------------------------------------- time: 00:28:23 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839544/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839544/ "2022-05-18T08:38:09.804065+0000 mon.a (mon.0) 1940 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6839547] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:55:30 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839547/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839547/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [6839552] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:55:28 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839552/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839552/ Command failed on smithi085 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [6839554] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 05:18:02 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839554/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839554/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0279a1b32cbe4ab7989eacc03c2773a9 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=88ea61a1dffa02901b0c13f117ac41997d8644bc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' Dead ================================================================= [6839535] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839535/ log: http://qa-proxy.ceph.com/teuthology/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839535/ hit max job timeout Pass ================================================================= [6839507] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:31:12 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839507/ [6839509] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} time: 00:42:00 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839509/ [6839510] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:14:16 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839510/ [6839511] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:22:51 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839511/ [6839512] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 01:08:14 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839512/ [6839513] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:31 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839513/ [6839516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 01:09:18 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839516/ [6839517] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:29 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839517/ [6839518] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:13:58 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839518/ [6839519] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:42:28 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839519/ [6839520] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:23:00 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839520/ [6839521] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:29:10 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839521/ [6839522] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:39:12 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839522/ [6839524] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:42 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839524/ [6839525] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 00:46:56 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839525/ [6839526] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:55 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839526/ [6839527] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:38:57 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839527/ [6839528] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:30:09 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839528/ [6839530] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:37 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839530/ [6839531] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:11:31 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839531/ [6839533] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:06:07 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839533/ [6839536] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:05 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839536/ [6839537] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:51:24 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839537/ [6839539] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:33:54 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839539/ [6839540] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} time: 00:49:45 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839540/ [6839541] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:46:29 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839541/ [6839542] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:30:08 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839542/ [6839543] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:37:21 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839543/ [6839545] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:54:24 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839545/ [6839546] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:18:17 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839546/ [6839548] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 00:59:22 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839548/ [6839549] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:53 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839549/ [6839550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:19:11 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839550/ [6839551] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:34 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839551/ [6839553] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:28 info: http://pulpito.ceph.com/nshelke-2022-05-18_07:35:50-fs-wip-nshelke-pools-testing-testing-default-smithi/6839553/