2023-02-22T15:55:37.237 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-22T15:55:37.237 INFO:teuthology.results:subset: '111/120000' 2023-02-22T15:55:37.237 INFO:teuthology.results:seed: '528' 2023-02-22T21:55:56.686 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-22T21:55:56.715 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-02-23T05:51:38.263 INFO:teuthology.results:Tests finished! gathering results... 2023-02-23T05:51:54.842 INFO:scrape:Found 184 jobs 2023-02-23T05:52:03.330 INFO:scrape:Found 10 distinct failure reasons 2023-02-23T05:52:03.331 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9e38deda4c64fb29f34c8a63ce044f8a594b27fa 2023-02-23T05:52:03.331 INFO:scrape:2 jobs: ['7182998', '7183098'] 2023-02-23T05:52:03.331 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-02-23T05:52:03.331 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-02-23T05:52:03.331 INFO:scrape: 2023-02-23T05:52:03.331 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2023-02-23T05:52:03.332 INFO:scrape:2 jobs: ['7182857', '7183114'] 2023-02-23T05:52:03.332 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/acls}'] 2023-02-23T05:52:03.332 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}'] 2023-02-23T05:52:03.332 INFO:scrape: 2023-02-23T05:52:03.332 INFO:scrape:Failure: "2023-02-22T20:26:26.037974+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 234717.338906 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-02-23T05:52:03.332 INFO:scrape:1 jobs: ['7183016'] 2023-02-23T05:52:03.332 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-02-23T05:52:03.333 INFO:scrape: 2023-02-23T05:52:03.333 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi136 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9e38deda4c64fb29f34c8a63ce044f8a594b27fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-02-23T05:52:03.333 INFO:scrape:1 jobs: ['7183011'] 2023-02-23T05:52:03.333 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-02-23T05:52:03.333 INFO:scrape: 2023-02-23T05:52:03.333 INFO:scrape:Failure: "2023-02-22T22:28:24.224357+0000 mds.d (mds.0) 1 : cluster [WRN] client.4678 isn't responding to mclientcaps(revoke), ino 0x100000059e5 pending pFc issued pFcb, sent 300.004575 seconds ago" in cluster log 2023-02-23T05:52:03.333 INFO:scrape:1 jobs: ['7183102'] 2023-02-23T05:52:03.333 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-02-23T05:52:03.333 INFO:scrape: 2023-02-23T05:52:03.333 INFO:scrape:Failure: Command failed on smithi085 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' 2023-02-23T05:52:03.333 INFO:scrape:2 jobs: ['7182932', '7182887'] 2023-02-23T05:52:03.334 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-02-23T05:52:03.334 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/asok_dump_tree}', 'tasks/workunit/snaps}'] 2023-02-23T05:52:03.334 INFO:scrape: 2023-02-23T05:52:03.334 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-02-23T05:52:03.334 INFO:scrape:1 jobs: ['7183055'] 2023-02-23T05:52:03.334 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-02-23T05:52:03.335 INFO:scrape: 2023-02-23T05:52:03.335 INFO:scrape:Failure: "2023-02-22T21:57:09.750527+0000 mon.a (mon.0) 530 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log 2023-02-23T05:52:03.335 INFO:scrape:3 jobs: ['7183080', '7182970', '7182865'] 2023-02-23T05:52:03.336 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health'] 2023-02-23T05:52:03.336 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/exports}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-02-23T05:52:03.336 INFO:scrape: 2023-02-23T05:52:03.337 INFO:scrape:Dead: b'2023-02-23T05:45:51.966 DEBUG:teuthology.exit:Finished running handlers' 2023-02-23T05:52:03.337 INFO:scrape:1 jobs: ['7182879'] 2023-02-23T05:52:03.337 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2023-02-23T05:52:03.337 INFO:scrape: 2023-02-23T05:52:03.338 INFO:scrape:Failure: Command failed on smithi188 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ba8fe12-b2d8-11ed-9ae8-001a4aab830c -- ceph mon dump -f json' 2023-02-23T05:52:03.338 INFO:scrape:1 jobs: ['7182869'] 2023-02-23T05:52:03.338 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-02-23T05:52:03.339 INFO:scrape: 2023-02-23T05:52:03.350 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/ failed: 14 dead: 1 running: 0 waiting: 0 queued: 0 passed: 169 Fail ================================================================= [7182857] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:18:24 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182857/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec0e016152844286b0c5881d57b4d53b Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7182865] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:08:55 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182865/ "2023-02-22T18:38:48.901980+0000 mon.a (mon.0) 4210 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [7182869] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:13:29 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182869/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182869/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a67bc033d94488b9c6ecf86696b121b Command failed on smithi188 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ba8fe12-b2d8-11ed-9ae8-001a4aab830c -- ceph mon dump -f json' [7182887] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182887/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd0971938c204d5c928add210fe0a159 Command failed on smithi037 with status 2: 'sudo dpkg -i /tmp/linux- image.deb' [7182932] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:05:33 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182932/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec814422c79c47458ddc0f624e8d9bd2 Command failed on smithi085 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' [7182970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:57:20 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182970/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182970/ "2023-02-22T20:14:17.332139+0000 mon.a (mon.0) 3677 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [7182998] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:04 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182998/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13e37f21124344158e4c80ac2d155600 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9e38deda4c64fb29f34c8a63ce044f8a594b27fa [7183011] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:25 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183011/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183011/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5893f45836d6445ebaa32dfd92b67f63 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi136 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9e38deda4c64fb29f34c8a63ce044f8a594b27fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7183016] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:22:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183016/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183016/ "2023-02-22T20:26:26.037974+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 234717.338906 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7183055] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:52:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183055/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183055/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1aebdfc426c748d1b90a58973e3b8ff0 Command failed (workunit test suites/fsstress.sh) on smithi067 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9e38deda4c64fb29f34c8a63ce044f8a594b27fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7183080] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:55:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183080/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183080/ "2023-02-22T21:57:09.750527+0000 mon.a (mon.0) 530 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log [7183098] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183098/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183098/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fce1f748abb4ce8ae1b697ce772153b Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9e38deda4c64fb29f34c8a63ce044f8a594b27fa [7183102] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:04:23 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183102/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183102/ "2023-02-22T22:28:24.224357+0000 mds.d (mds.0) 1 : cluster [WRN] client.4678 isn't responding to mclientcaps(revoke), ino 0x100000059e5 pending pFc issued pFcb, sent 300.004575 seconds ago" in cluster log [7183114] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:13:14 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183114/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183114/ sentry: https://sentry.ceph.com/organizations/ceph/?query=584c7d58b9aa4624824d56eb97547000 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) Dead ================================================================= [7182879] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182879/ hit max job timeout Pass ================================================================= [7182801] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:19 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182801/ [7182803] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:22:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182803/ [7182805] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:31:44 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182805/ [7182807] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:24:17 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182807/ [7182809] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:32:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182809/ [7182811] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182811/ [7182813] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:32:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182813/ [7182815] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:45:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182815/ [7182817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:31:23 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182817/ [7182819] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:31:45 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182819/ [7182821] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:33:26 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182821/ [7182823] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:28:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182823/ [7182825] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 01:08:47 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182825/ [7182827] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:31:50 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182827/ [7182829] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182829/ [7182831] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:19:35 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182831/ [7182833] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:45 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182833/ [7182835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:19:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182835/ [7182837] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:40:56 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182837/ [7182839] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:22:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182839/ [7182841] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:47 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182841/ [7182843] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182843/ [7182845] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:17:01 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182845/ [7182847] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 02:00:59 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182847/ [7182849] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:40:25 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182849/ [7182851] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182851/ [7182853] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:27 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182853/ [7182855] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:21:14 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182855/ [7182859] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182859/ [7182861] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:58:47 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182861/ [7182863] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:02:39 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182863/ [7182867] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:11:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182867/ [7182871] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:53:29 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182871/ [7182873] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:25:25 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182873/ [7182875] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182875/ [7182877] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182877/ [7182881] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:12:21 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182881/ [7182883] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:40 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182883/ [7182885] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:30:37 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182885/ [7182889] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:54 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182889/ [7182891] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:13:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182891/ [7182893] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:23:40 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182893/ [7182895] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:44:31 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182895/ [7182897] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182897/ [7182899] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:27:23 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182899/ [7182901] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:24:47 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182901/ [7182903] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182903/ [7182905] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:50:32 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182905/ [7182907] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:56 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182907/ [7182909] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182909/ [7182911] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:49:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182911/ [7182913] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:19:24 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182913/ [7182915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:21:35 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182915/ [7182917] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:12:51 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182917/ [7182919] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 01:56:35 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182919/ [7182921] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:22:30 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182921/ [7182923] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:41:52 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182923/ [7182925] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:48:43 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182925/ [7182926] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:20 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182926/ [7182928] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182928/ [7182930] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:35:50 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182930/ [7182934] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:12:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182934/ [7182936] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:20:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182936/ [7182938] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:52 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182938/ [7182940] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:08:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182940/ [7182942] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:19:51 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182942/ [7182945] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:04:36 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182945/ [7182947] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:11:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182947/ [7182949] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:52:01 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182949/ [7182951] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:12:44 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182951/ [7182953] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:21 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182953/ [7182955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:40:31 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182955/ [7182957] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:29 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182957/ [7182958] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:21:55 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182958/ [7182960] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:12:13 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182960/ [7182963] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:27:15 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182963/ [7182964] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:27:30 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182964/ [7182966] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:28:43 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182966/ [7182968] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182968/ [7182972] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:39 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182972/ [7182974] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182974/ [7182976] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182976/ [7182978] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:30:25 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182978/ [7182980] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182980/ [7182982] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182982/ [7182984] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:15:34 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182984/ [7182986] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:19:27 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182986/ [7182989] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182989/ [7182990] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:32 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182990/ [7182992] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:55:51 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182992/ [7182994] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:29:49 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182994/ [7182996] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:20:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7182996/ [7183000] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 01:00:30 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183000/ [7183002] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:31 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183002/ [7183005] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:25:08 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183005/ [7183007] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:38:40 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183007/ [7183009] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:11:19 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183009/ [7183013] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:27:16 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183013/ [7183015] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:21:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183015/ [7183019] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:35:15 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183019/ [7183021] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:44 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183021/ [7183023] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:11:23 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183023/ [7183025] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:16 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183025/ [7183027] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:18:12 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183027/ [7183029] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:57 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183029/ [7183031] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:24:53 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183031/ [7183033] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:46:50 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183033/ [7183035] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:28:45 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183035/ [7183037] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183037/ [7183039] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:14 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183039/ [7183041] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:18:43 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183041/ [7183043] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:17:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183043/ [7183045] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183045/ [7183047] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:33:53 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183047/ [7183049] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:49:50 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183049/ [7183051] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:17:37 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183051/ [7183053] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:32 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183053/ [7183057] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:40:26 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183057/ [7183059] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:11:56 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183059/ [7183061] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:29:58 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183061/ [7183063] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:56 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183063/ [7183064] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:06:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183064/ [7183065] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:33 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183065/ [7183066] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:00:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183066/ [7183067] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:03 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183067/ [7183068] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:22 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183068/ [7183069] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:03 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183069/ [7183070] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183070/ [7183071] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:39:44 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183071/ [7183072] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:47 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183072/ [7183073] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:53:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183073/ [7183074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:23:27 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183074/ [7183075] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:09 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183075/ [7183076] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:25:17 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183076/ [7183077] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183077/ [7183078] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:23:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183078/ [7183079] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:28:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183079/ [7183081] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:15 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183081/ [7183082] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:16 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183082/ [7183083] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:26 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183083/ [7183084] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:21:02 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183084/ [7183085] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:13:59 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183085/ [7183086] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183086/ [7183087] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:57:05 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183087/ [7183088] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:59:42 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183088/ [7183089] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:00 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183089/ [7183090] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:06:04 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183090/ [7183091] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:09:10 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183091/ [7183092] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:25:06 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183092/ [7183093] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183093/ [7183094] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:11 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183094/ [7183095] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:32:33 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183095/ [7183096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:42:52 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183096/ [7183097] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:20:59 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183097/ [7183099] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:34 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183099/ [7183100] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:12:50 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183100/ [7183101] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:30 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183101/ [7183103] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:13:18 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183103/ [7183104] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:14:14 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183104/ [7183105] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:29:34 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183105/ [7183106] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:55 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183106/ [7183107] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:36:07 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183107/ [7183108] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:45:19 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183108/ [7183109] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:46 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183109/ [7183110] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:45 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183110/ [7183111] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:35:15 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183111/ [7183112] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:15:30 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183112/ [7183113] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:29:28 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183113/ [7183115] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:03 info: http://pulpito.ceph.com/yuriw-2023-02-21_19:52:51-fs-wip-yuri3-testing-2023-02-21-0727-quincy-distro-default-smithi/7183115/