2022-04-08T07:38:28.099 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-08T07:38:28.101 INFO:teuthology.results:subset: '21/32' 2022-04-08T07:38:28.101 INFO:teuthology.results:seed: '6329' 2022-04-08T09:44:53.233 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-08T09:44:53.263 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-08T16:00:21.001 INFO:teuthology.results:Tests finished! gathering results... 2022-04-08T16:00:25.185 INFO:scrape:Found 50 jobs 2022-04-08T16:00:33.561 INFO:scrape:Found 13 distinct failure reasons 2022-04-08T16:00:33.562 INFO:scrape:Failure: Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2022-04-08T16:00:33.562 INFO:scrape:1 jobs: ['6782641'] 2022-04-08T16:00:33.563 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/metrics}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.563 INFO:scrape: 2022-04-08T16:00:33.564 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-04-08T16:00:33.564 INFO:scrape:1 jobs: ['6782623'] 2022-04-08T16:00:33.565 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.565 INFO:scrape: 2022-04-08T16:00:33.565 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1649409310.690:202): avc: denied { node_bind } for pid=1506 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-04-08T16:00:33.566 INFO:scrape:1 jobs: ['6782643'] 2022-04-08T16:00:33.566 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.567 INFO:scrape: 2022-04-08T16:00:33.567 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2022-04-08T16:00:33.568 INFO:scrape:3 jobs: ['6782650', '6782625', '6782617'] 2022-04-08T16:00:33.568 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'races', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.569 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.569 INFO:scrape: 2022-04-08T16:00:33.570 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status 2022-04-08T16:00:33.570 INFO:scrape:1 jobs: ['6782630'] 2022-04-08T16:00:33.570 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}'] 2022-04-08T16:00:33.571 INFO:scrape: 2022-04-08T16:00:33.571 INFO:scrape:Failure: Command failed on smithi092 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/vshankar/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 65c3be207e31257427bd5247ff8df18642999daa' 2022-04-08T16:00:33.572 INFO:scrape:1 jobs: ['6782622'] 2022-04-08T16:00:33.572 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsx}', 'wsync/{no}}'] 2022-04-08T16:00:33.573 INFO:scrape: 2022-04-08T16:00:33.573 INFO:scrape:Failure: "2022-04-08T10:01:53.305489+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-08T16:00:33.573 INFO:scrape:1 jobs: ['6782651'] 2022-04-08T16:00:33.574 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.575 INFO:scrape: 2022-04-08T16:00:33.575 INFO:scrape:Dead: b'2022-04-08T14:30:22.551 DEBUG:teuthology.exit:Finished running handlers' 2022-04-08T16:00:33.575 INFO:scrape:6 jobs: ['6782608', '6782642', '6782646', '6782640', '6782607', '6782633'] 2022-04-08T16:00:33.576 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.576 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/fs/snaps}}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{pg-warn', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'roles', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2022-04-08T16:00:33.577 INFO:scrape: 2022-04-08T16:00:33.577 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh 2022-04-08T16:00:33.578 INFO:scrape:1 jobs: ['6782620'] 2022-04-08T16:00:33.578 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{yes}}'] 2022-04-08T16:00:33.579 INFO:scrape: 2022-04-08T16:00:33.579 INFO:scrape:Failure: Command failed on smithi120 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7b13d4ec-b71e-11ec-8c36-001a4aab830c -- ceph mon dump -f json' 2022-04-08T16:00:33.579 INFO:scrape:1 jobs: ['6782648'] 2022-04-08T16:00:33.580 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.580 INFO:scrape: 2022-04-08T16:00:33.581 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-04-08T16:00:33.581 INFO:scrape:1 jobs: ['6782644'] 2022-04-08T16:00:33.582 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}'] 2022-04-08T16:00:33.582 INFO:scrape: 2022-04-08T16:00:33.583 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2022-04-08T16:00:33.583 INFO:scrape:1 jobs: ['6782628'] 2022-04-08T16:00:33.583 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/acls}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.584 INFO:scrape: 2022-04-08T16:00:33.584 INFO:scrape:Failure: "2022-04-08T09:50:53.130761+0000 mds.a (mds.0) 1 : cluster [WRN] client.4684 isn't responding to mclientcaps(revoke), ino 0x1000000485e pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.030500 seconds ago" in cluster log 2022-04-08T16:00:33.585 INFO:scrape:1 jobs: ['6782645'] 2022-04-08T16:00:33.585 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-08T16:00:33.585 INFO:scrape: 2022-04-08T16:00:33.588 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/ failed: 13 dead: 7 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [6782617] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 05:16:24 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782617/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fd447f9446b404a8fda2e81080a6802 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi022 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=65c3be207e31257427bd5247ff8df18642999daa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6782620] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 03:28:06 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782620/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27e3902eeb1b4e38bf7d811b1110a454 Command failed (workunit test suites/ffsb.sh) on smithi073 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=65c3be207e31257427bd5247ff8df18642999daa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [6782622] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:24:11 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782622/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=278554e025be4bceb733b695ccd367fb Command failed on smithi092 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/vshankar/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 65c3be207e31257427bd5247ff8df18642999daa' [6782625] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 06:19:16 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782625/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40370eb0d9dc444f83a83523ef472d73 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi082 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=65c3be207e31257427bd5247ff8df18642999daa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6782628] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} ----------------------------------------------------------------- time: 00:13:23 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782628/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d98fd8eee1b547aab24fb0e61faf88d8 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [6782630] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 01:03:23 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782630/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782630/ error during scrub thrashing: Command failed on smithi109 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [6782641] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} ----------------------------------------------------------------- time: 00:20:11 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782641/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782641/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc6cfceffb574123b992f2324158edf8 Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [6782643] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:23:54 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782643/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782643/ SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1649409310.690:202): avc: denied { node_bind } for pid=1506 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6782644] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 01:52:35 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782644/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782644/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6782645] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:21:46 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782645/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782645/ "2022-04-08T09:50:53.130761+0000 mds.a (mds.0) 1 : cluster [WRN] client.4684 isn't responding to mclientcaps(revoke), ino 0x1000000485e pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.030500 seconds ago" in cluster log [6782648] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782648/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782648/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43de964a6f47465d93f69862aa43e29f Command failed on smithi120 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7b13d4ec-b71e-11ec-8c36-001a4aab830c -- ceph mon dump -f json' [6782650] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 05:07:49 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782650/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6fae368617e4af39019dca0abeaffe2 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=65c3be207e31257427bd5247ff8df18642999daa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6782651] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:45:55 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782651/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782651/ "2022-04-08T10:01:53.305489+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [6782607] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782607/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782607/ hit max job timeout [6782608] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782608/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782608/ hit max job timeout [6782623] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:34:28 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782623/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fd991b43f8041c3a9a77ed1579e3a98 Command failed (workunit test suites/fsstress.sh) on smithi173 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=65c3be207e31257427bd5247ff8df18642999daa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6782633] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782633/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782633/ hit max job timeout [6782640] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782640/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782640/ hit max job timeout [6782642] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782642/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782642/ hit max job timeout [6782646] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782646/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782646/ hit max job timeout Pass ================================================================= [6782602] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:28:18 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782602/ [6782603] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 01:06:06 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782603/ [6782604] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:49:37 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782604/ [6782605] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:29:01 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782605/ [6782606] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:46:52 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782606/ [6782609] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:35:48 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782609/ [6782610] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:56 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782610/ [6782611] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:53:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782611/ [6782612] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 02:02:50 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782612/ [6782613] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:26:28 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782613/ [6782614] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:34:56 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782614/ [6782615] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:52:09 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782615/ [6782616] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:54:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782616/ [6782618] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:37:58 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782618/ [6782619] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:20:12 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782619/ [6782621] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:32:52 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782621/ [6782624] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:30:02 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782624/ [6782626] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:36:12 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782626/ [6782627] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} time: 00:32:33 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782627/ [6782629] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:23:56 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782629/ [6782631] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 02:24:00 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782631/ [6782632] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:07:42 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782632/ [6782634] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:37 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782634/ [6782635] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 01:07:18 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782635/ [6782636] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:30:39 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782636/ [6782637] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:39:03 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782637/ [6782638] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:35:02 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782638/ [6782639] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:35:40 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782639/ [6782647] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:47 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782647/ [6782649] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:28:09 info: http://pulpito.ceph.com/vshankar-2022-04-08_07:36:22-fs-wip-vshankar-testing-55110-20220407-173953-testing-default-smithi/6782649/