2022-04-27T05:36:22.898 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T05:36:22.899 INFO:teuthology.results:subset: '7/32' 2022-04-27T05:36:22.899 INFO:teuthology.results:seed: '6004' 2022-04-27T06:01:36.892 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T06:01:36.918 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-27T12:35:07.773 INFO:teuthology.results:Tests finished! gathering results... 2022-04-27T12:35:11.404 INFO:scrape:Found 40 jobs 2022-04-27T12:35:17.972 INFO:scrape:Found 8 distinct failure reasons 2022-04-27T12:35:17.973 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2022-04-27T12:35:17.973 INFO:scrape:1 jobs: ['6808547'] 2022-04-27T12:35:17.973 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.973 INFO:scrape: 2022-04-27T12:35:17.973 INFO:scrape:Failure: "2022-04-27T06:08:09.324965+0000 mon.a (mon.0) 461 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-27T12:35:17.973 INFO:scrape:1 jobs: ['6808539'] 2022-04-27T12:35:17.974 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.974 INFO:scrape: 2022-04-27T12:35:17.974 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-04-27T12:35:17.974 INFO:scrape:2 jobs: ['6808535', '6808528'] 2022-04-27T12:35:17.974 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.974 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.974 INFO:scrape: 2022-04-27T12:35:17.975 INFO:scrape:Failure: Extra data: line 2 column 557 (char 557) 2022-04-27T12:35:17.975 INFO:scrape:2 jobs: ['6808529', '6808548'] 2022-04-27T12:35:17.975 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.975 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.975 INFO:scrape: 2022-04-27T12:35:17.975 INFO:scrape:Dead: b'2022-04-27T12:20:03.395 DEBUG:teuthology.exit:Finished running handlers' 2022-04-27T12:35:17.975 INFO:scrape:1 jobs: ['6808527'] 2022-04-27T12:35:17.976 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2022-04-27T12:35:17.976 INFO:scrape: 2022-04-27T12:35:17.976 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-04-27T12:35:17.976 INFO:scrape:2 jobs: ['6808552', '6808526'] 2022-04-27T12:35:17.976 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}'] 2022-04-27T12:35:17.976 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-27T12:35:17.976 INFO:scrape: 2022-04-27T12:35:17.976 INFO:scrape:Failure: "2022-04-27T06:54:00.642242+0000 mds.e (mds.3) 1 : cluster [WRN] client.4658 isn't responding to mclientcaps(revoke), ino 0x1000000d26f pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.197299 seconds ago" in cluster log 2022-04-27T12:35:17.977 INFO:scrape:2 jobs: ['6808553', '6808519'] 2022-04-27T12:35:17.977 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_entire_dirfrags/yes', 'races', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.977 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.977 INFO:scrape: 2022-04-27T12:35:17.977 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-04-27T12:35:17.977 INFO:scrape:1 jobs: ['6808544'] 2022-04-27T12:35:17.978 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T12:35:17.978 INFO:scrape: 2022-04-27T12:35:17.980 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/ failed: 11 dead: 1 running: 0 waiting: 0 queued: 0 passed: 28 Fail ================================================================= [6808519] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:10:22 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808519/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808519/ "2022-04-27T06:11:05.297331+0000 mds.f (mds.0) 1 : cluster [WRN] client.4577 isn't responding to mclientcaps(revoke), ino 0x10000004c0a pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.143370 seconds ago" in cluster log [6808526] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 01:53:07 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808526/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808526/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6808528] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:52 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808528/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0c5e82a045b4fa4a73befe7bdf479cc Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi185 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6808529] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:25:50 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808529/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25f7a5f33edb47da997816159f346e3d Extra data: line 2 column 557 (char 557) [6808535] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:22:10 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808535/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808535/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eec95ca00d364d6091d20f63bbf24f48 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6808539] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:46:53 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808539/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808539/ "2022-04-27T06:08:09.324965+0000 mon.a (mon.0) 461 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6808544] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:15:58 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808544/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808544/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a21760f3c32040fa9c13428b2e633dd9 Command failed (workunit test suites/pjd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6808547] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 05:31:47 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808547/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=297e24ce0e7e453fadb0e7ad23b0d536 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi061 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6808548] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:23:32 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808548/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808548/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffe342d180c74c45ab29d6d3a7c31953 Extra data: line 2 column 555 (char 555) [6808552] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 01:15:06 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808552/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808552/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6808553] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:52:31 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808553/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808553/ "2022-04-27T06:54:00.642242+0000 mds.e (mds.3) 1 : cluster [WRN] client.4658 isn't responding to mclientcaps(revoke), ino 0x1000000d26f pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.197299 seconds ago" in cluster log Dead ================================================================= [6808527] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808527/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808527/ hit max job timeout Pass ================================================================= [6808518] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:31 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808518/ [6808520] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:02:24 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808520/ [6808521] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:10:02 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808521/ [6808522] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:35:32 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808522/ [6808523] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:29:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808523/ [6808524] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:51:54 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808524/ [6808525] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} time: 00:43:55 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808525/ [6808530] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:13:51 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808530/ [6808531] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:48:05 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808531/ [6808532] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:05 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808532/ [6808533] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:59:30 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808533/ [6808534] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:40 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808534/ [6808536] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:58 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808536/ [6808537] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:32 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808537/ [6808538] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:35:02 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808538/ [6808540] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:10:12 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808540/ [6808541] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:12:07 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808541/ [6808542] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:08 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808542/ [6808543] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:38:55 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808543/ [6808545] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:29 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808545/ [6808546] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:32:33 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808546/ [6808549] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:20 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808549/ [6808550] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:29:42 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808550/ [6808551] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:17:34 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808551/ [6808554] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:43:36 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808554/ [6808555] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:17 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808555/ [6808556] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:20:33 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808556/ [6808557] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:31:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_05:35:50-fs-wip-55170-20220425-134100-distro-default-smithi/6808557/