2023-02-10T18:16:46.060 INFO:root:teuthology version: 0.0.1.dev37+g00690ec 2023-02-10T18:16:46.060 INFO:teuthology.results:subset: '111/120000' 2023-02-10T18:16:46.060 INFO:teuthology.results:seed: '6087' 2023-02-10T20:34:06.260 INFO:root:teuthology version: 0.0.1.dev37+g00690ec 2023-02-10T20:34:06.369 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-02-11T07:21:09.643 INFO:teuthology.results:Tests finished! gathering results... 2023-02-11T07:22:06.923 INFO:scrape:Found 184 jobs 2023-02-11T07:23:10.552 INFO:scrape:Found 13 distinct failure reasons 2023-02-11T07:23:10.566 INFO:scrape:Dead: b'2023-02-11T06:55:57.066 DEBUG:teuthology.exit:Finished running handlers' 2023-02-11T07:23:10.566 INFO:scrape:2 jobs: ['7155668', '7155699'] 2023-02-11T07:23:10.566 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-02-11T07:23:10.566 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', 'begin/{0-install', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'debug', 'fs/valgrind/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/whitelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/mirror}}', 'tasks/{0-check-counter', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2023-02-11T07:23:10.566 INFO:scrape: 2023-02-11T07:23:10.566 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi143 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-02-11T07:23:10.566 INFO:scrape:8 jobs: ['7155701', '7155763', '7155810', '7155804', '7155665', '7155671', '7155738', '7155745'] 2023-02-11T07:23:10.567 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'mds', 'mon', 'osd}', 'overrides/{frag'] 2023-02-11T07:23:10.567 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/ffsb}}', '2-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'rhel_8}', 'scrub/no', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'traceless/50pc}', 'workunit/suites/ffsb}', 'wsync/{yes}}'] 2023-02-11T07:23:10.567 INFO:scrape: 2023-02-11T07:23:10.567 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-02-11T07:23:10.567 INFO:scrape:11 jobs: ['7155631', '7155753', '7155662', '7155725', '7155799', '7155734', '7155787', '7155695', '7155660', '7155759', '7155681'] 2023-02-11T07:23:10.567 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-02-11T07:23:10.568 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-02-11T07:23:10.568 INFO:scrape: 2023-02-11T07:23:10.568 INFO:scrape:Failure: Test failure: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2023-02-11T07:23:10.568 INFO:scrape:1 jobs: ['7155724'] 2023-02-11T07:23:10.568 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}'] 2023-02-11T07:23:10.568 INFO:scrape: 2023-02-11T07:23:10.568 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9839dd56836c942b9185baee816ddb2c7166f950 2023-02-11T07:23:10.568 INFO:scrape:2 jobs: ['7155795', '7155728'] 2023-02-11T07:23:10.568 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-02-11T07:23:10.568 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-02-11T07:23:10.568 INFO:scrape: 2023-02-11T07:23:10.569 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-02-11T07:23:10.569 INFO:scrape:2 jobs: ['7155788', '7155661'] 2023-02-11T07:23:10.569 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-02-11T07:23:10.569 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-02-11T07:23:10.569 INFO:scrape: 2023-02-11T07:23:10.569 INFO:scrape:Failure: "2023-02-10T18:55:36.427170+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1894 objects degraded (0.053%), 1 pg degraded (PG_DEGRADED)" in cluster log 2023-02-11T07:23:10.569 INFO:scrape:1 jobs: ['7155637'] 2023-02-11T07:23:10.569 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-02-11T07:23:10.569 INFO:scrape: 2023-02-11T07:23:10.569 INFO:scrape:Failure: "2023-02-10T19:50:26.064563+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 445283.622661 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-02-11T07:23:10.569 INFO:scrape:1 jobs: ['7155737'] 2023-02-11T07:23:10.570 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-02-11T07:23:10.570 INFO:scrape: 2023-02-11T07:23:10.570 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-02-11T07:23:10.570 INFO:scrape:2 jobs: ['7155718', '7155756'] 2023-02-11T07:23:10.570 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{frag', 'ranks/5'] 2023-02-11T07:23:10.570 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'scrub/no', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/suites/fsstress}', 'wsync/{yes}}'] 2023-02-11T07:23:10.570 INFO:scrape: 2023-02-11T07:23:10.570 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2023-02-11T07:23:10.570 INFO:scrape:2 jobs: ['7155811', '7155657'] 2023-02-11T07:23:10.570 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/acls}'] 2023-02-11T07:23:10.570 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}'] 2023-02-11T07:23:10.571 INFO:scrape: 2023-02-11T07:23:10.571 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi005 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' 2023-02-11T07:23:10.571 INFO:scrape:1 jobs: ['7155676'] 2023-02-11T07:23:10.571 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/dbench}', 'wsync/{yes}}'] 2023-02-11T07:23:10.571 INFO:scrape: 2023-02-11T07:23:10.571 INFO:scrape:Failure: "2023-02-10T20:52:22.707040+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi167:0 (7375), after waiting 48.4702 seconds during MDS startup" in cluster log 2023-02-11T07:23:10.571 INFO:scrape:1 jobs: ['7155793'] 2023-02-11T07:23:10.571 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-02-11T07:23:10.571 INFO:scrape: 2023-02-11T07:23:10.571 INFO:scrape:Failure: 2023-02-11T07:23:10.571 INFO:scrape:1 jobs: ['7155732'] 2023-02-11T07:23:10.571 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/1', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/fsx}', 'wsync/{yes}}'] 2023-02-11T07:23:10.572 INFO:scrape: 2023-02-11T07:23:10.586 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/ failed: 33 dead: 2 running: 0 waiting: 0 queued: 0 passed: 149 Fail ================================================================= [7155631] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155631/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155631/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2def5562b9247f2970c742f1ae5d321 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155637] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:29:41 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155637/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155637/ "2023-02-10T18:55:36.427170+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1894 objects degraded (0.053%), 1 pg degraded (PG_DEGRADED)" in cluster log [7155657] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:19:56 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155657/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bee5ce3d1ed648e19dbecc917d5fb5c4 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7155660] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155660/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155660/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ad9151982f74a10b08d2778161c521c Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155661] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:02:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155661/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ceae36535f754769aca773bb25647552 Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7155662] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155662/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155662/ sentry: https://sentry.ceph.com/organizations/ceph/?query=486438a955a942338f180c8c5ebd834b Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi162 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155665] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:13:20 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155665/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aff31abb45d449bfae0b9e724a3f4c8b Command failed (workunit test suites/ffsb.sh) on smithi097 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155671] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:16:21 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155671/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce12268dc40541019747d1114231e15e Command failed (workunit test suites/ffsb.sh) on smithi155 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155676] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 01:56:49 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155676/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155676/ error during scrub thrashing: Command failed on smithi005 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7155681] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:25:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155681/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155681/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93602d8e2b184428969bca8e85fb87bc Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi093 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155695] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155695/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77319f6149de46a2864edaf3af90d4d3 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi122 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155701] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:24:12 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155701/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155701/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ede3a51aea94ba09cf0ce67780d7888 Command failed (workunit test suites/ffsb.sh) on smithi143 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155718] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} ----------------------------------------------------------------- time: 06:31:17 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155718/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b03c6efb2594e4b87fd785bc9f34040 Command failed (workunit test suites/fsstress.sh) on smithi104 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7155724] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:17:51 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155724/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155724/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51b92996293d4bd297ed6ffa96f509ea Test failure: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) [7155725] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:15:31 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155725/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f62a70c96d647459d685d1edf1c0318 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155728] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:14 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155728/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1f5c59df2d14af79bca3f9a288d1c35 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9839dd56836c942b9185baee816ddb2c7166f950 [7155732] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} ----------------------------------------------------------------- time: 00:22:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155732/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f81b1ef9fe5471c9ab6198f1cb4452c [7155734] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:18:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155734/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e871d97cae674a8fbc167c54ed0bbd74 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi148 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155737] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:21:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155737/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155737/ "2023-02-10T19:50:26.064563+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 445283.622661 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7155738] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155738/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=051eadf3081f4d98b969492fdf3e9f59 Command failed (workunit test suites/ffsb.sh) on smithi162 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155745] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:15:56 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c3ffe0369ac4646ae987674cb449de1 Command failed (workunit test suites/ffsb.sh) on smithi120 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155753] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:18:03 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155753/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4139195c6f644425b5d9a6c6ff0c0496 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155756] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:50:17 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155756/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de4ec2080a7a44e7aa555bf5dd909a3f Command failed (workunit test suites/fsstress.sh) on smithi141 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7155759] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:26:13 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155759/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13aed9ce037b4aa59e121b4ff78c32c9 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155763] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155763/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad42a7d0999f4a02949ef2c5b40ab537 Command failed (workunit test suites/ffsb.sh) on smithi064 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155787] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:16:19 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be42b85fed5340fbaaad095e1873172b Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi012 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155788] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:54:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155788/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7715273bcbfb4cac84a34633c827b7e1 Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7155793] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:43:30 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155793/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155793/ "2023-02-10T20:52:22.707040+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi167:0 (7375), after waiting 48.4702 seconds during MDS startup" in cluster log [7155795] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83cf3120251d4ddb8f423c1649db0323 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9839dd56836c942b9185baee816ddb2c7166f950 [7155799] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:20:25 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155799/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4128bcdd24b4acaa7d9c616906eb1cf Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi133 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7155804] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155804/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38750b59dedb4940aaf95f421bbc2501 Command failed (workunit test suites/ffsb.sh) on smithi138 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh' [7155810] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:18:14 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18e1453c39ea4280923b449012b111bf Command failed (workunit test suites/ffsb.sh) on smithi169 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9839dd56836c942b9185baee816ddb2c7166f950 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7155811] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:12:17 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155811/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bb4cbfd3dd144df955b78ab3c64a9b8 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) Dead ================================================================= [7155668] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155668/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155668/ hit max job timeout [7155699] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155699/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155699/ hit max job timeout Pass ================================================================= [7155629] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:47 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155629/ [7155630] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:23:12 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155630/ [7155632] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:30:16 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155632/ [7155633] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155633/ [7155634] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:03 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155634/ [7155635] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:32:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155635/ [7155636] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155636/ [7155638] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:26:47 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155638/ [7155639] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:23:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155639/ [7155640] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:18:01 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155640/ [7155641] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 00:55:14 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155641/ [7155642] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:25:23 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155642/ [7155643] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155643/ [7155644] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:13:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155644/ [7155645] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155645/ [7155646] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:17:58 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155646/ [7155647] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:07 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155647/ [7155648] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:27 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155648/ [7155649] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155649/ [7155650] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155650/ [7155651] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:13:27 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155651/ [7155652] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} time: 02:22:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155652/ [7155653] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:26:05 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155653/ [7155654] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155654/ [7155655] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:03 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155655/ [7155656] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:20:01 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155656/ [7155658] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155658/ [7155659] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:57:20 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155659/ [7155663] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:35:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155663/ [7155664] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:54:01 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155664/ [7155666] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:53 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155666/ [7155667] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155667/ [7155669] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:12:36 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155669/ [7155670] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:09 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155670/ [7155672] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:28:23 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155672/ [7155673] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:56 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155673/ [7155674] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:12:43 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155674/ [7155675] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:24:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155675/ [7155677] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155677/ [7155678] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:25:17 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155678/ [7155679] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:35:20 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155679/ [7155680] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:19:16 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155680/ [7155682] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:50 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155682/ [7155683] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:13:21 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155683/ [7155684] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:48:33 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155684/ [7155685] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:20:35 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155685/ [7155686] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:21:41 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155686/ [7155687] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:12:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155687/ [7155688] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 01:57:57 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155688/ [7155689] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:27:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155689/ [7155690] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:42:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155690/ [7155691] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:50:01 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155691/ [7155692] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:30:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155692/ [7155693] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:22 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155693/ [7155694] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:39:35 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155694/ [7155696] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:10:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155696/ [7155697] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:20:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155697/ [7155698] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:56 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155698/ [7155700] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:20:02 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155700/ [7155702] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:12:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155702/ [7155703] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:56:46 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155703/ [7155704] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:12:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155704/ [7155705] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:16 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155705/ [7155706] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:36:42 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155706/ [7155707] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:35 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155707/ [7155708] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:21:40 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155708/ [7155709] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:11:59 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155709/ [7155710] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:27:31 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155710/ [7155711] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:24:35 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155711/ [7155712] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:29:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155712/ [7155713] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:18 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155713/ [7155714] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:54:49 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155714/ [7155715] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:28 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155715/ [7155716] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:22 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155716/ [7155717] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:04 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155717/ [7155719] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:15:28 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155719/ [7155720] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155720/ [7155721] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:15 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155721/ [7155722] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:18:26 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155722/ [7155723] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:14 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155723/ [7155726] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155726/ [7155727] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:21:06 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155727/ [7155729] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 01:05:50 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155729/ [7155730] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155730/ [7155731] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:25:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155731/ [7155733] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:12:43 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155733/ [7155735] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:26:51 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155735/ [7155736] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:20:27 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155736/ [7155739] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155739/ [7155740] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:12:32 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155740/ [7155741] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155741/ [7155742] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:18:08 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155742/ [7155743] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:46 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155743/ [7155744] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:25:33 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155744/ [7155746] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:29:44 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155746/ [7155747] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:09 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155747/ [7155748] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155748/ [7155749] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:18:39 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155749/ [7155750] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:20:13 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155750/ [7155751] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:22 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155751/ [7155752] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:33:27 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155752/ [7155754] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:00:03 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155754/ [7155755] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:20 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155755/ [7155757] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:39:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155757/ [7155758] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:12:14 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155758/ [7155760] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155760/ [7155761] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:00:37 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155761/ [7155762] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:04 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155762/ [7155764] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:52 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155764/ [7155765] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155765/ [7155766] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:46 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155766/ [7155767] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:19:06 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155767/ [7155768] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:39:05 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155768/ [7155769] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:43 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155769/ [7155770] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:51:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155770/ [7155771] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:20:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155771/ [7155772] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:02 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155772/ [7155773] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:27:31 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155773/ [7155774] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:14:53 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155774/ [7155775] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:23:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155775/ [7155776] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:27:33 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155776/ [7155777] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:55:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155777/ [7155778] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155778/ [7155779] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:15 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155779/ [7155780] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:04 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155780/ [7155781] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:21:19 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155781/ [7155782] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155782/ [7155783] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:13:56 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155783/ [7155784] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:50:32 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155784/ [7155785] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:58:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155785/ [7155786] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155786/ [7155789] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:22:42 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155789/ [7155790] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155790/ [7155791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:15:28 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155791/ [7155792] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:32:15 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155792/ [7155794] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:42 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155794/ [7155796] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:54 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155796/ [7155797] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155797/ [7155798] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:17:27 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155798/ [7155800] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155800/ [7155801] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:44 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155801/ [7155802] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:28:55 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155802/ [7155803] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:28 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155803/ [7155805] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:47:07 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155805/ [7155806] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:00 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155806/ [7155807] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155807/ [7155808] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:35:38 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155808/ [7155809] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155809/ [7155812] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:10 info: http://pulpito.ceph.com/yuriw-2023-02-07_20:51:00-fs-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7155812/