2024-04-11T06:35:08.277 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-11T06:35:08.277 INFO:teuthology.results:subset: '629/1024' 2024-04-11T06:35:08.277 INFO:teuthology.results:seed: '7834' 2024-04-11T09:46:47.592 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-11T09:46:47.615 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-11T20:17:44.670 INFO:teuthology.results:Tests finished! gathering results... 2024-04-11T20:18:03.949 INFO:scrape:Found 205 jobs 2024-04-11T20:18:42.869 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651763/teuthology.log 2024-04-11T20:18:44.226 INFO:scrape:Found 20 distinct failure reasons 2024-04-11T20:18:44.226 INFO:scrape:Failure: failed to install new kernel version within timeout 2024-04-11T20:18:44.227 INFO:scrape:1 jobs: ['7651797'] 2024-04-11T20:18:44.227 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-04-11T20:18:44.227 INFO:scrape: 2024-04-11T20:18:44.227 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi139.front.sepia.ceph.com: ['type=AVC msg=audit(1712825603.704:207): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-04-11T20:18:44.227 INFO:scrape:1 jobs: ['7651837'] 2024-04-11T20:18:44.227 INFO:scrape:suites: ['1-rename', 'distro/single-container-host', 'fs/cephadm/renamevolume/{0-start', 'overrides/{ignorelist_health', 'pg_health}}'] 2024-04-11T20:18:44.227 INFO:scrape: 2024-04-11T20:18:44.227 INFO:scrape:Failure: "2024-04-11T08:53:13.825239+0000 mds.f (mds.0) 1 : cluster [WRN] client.4663 isn't responding to mclientcaps(revoke), ino 0x100000030b0 pending pAsLsXsFcb issued pAsxLsXsxFscb, sent 301.564764 seconds ago" in cluster log 2024-04-11T20:18:44.227 INFO:scrape:3 jobs: ['7651769', '7651859', '7651780'] 2024-04-11T20:18:44.228 INFO:scrape:suites intersection: ['2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}'] 2024-04-11T20:18:44.228 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'export-check', 'frag', 'fs/functional/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'n/3', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'ranks/3', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v1', 'tasks/forward-scrub}', 'tasks/fsstress', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-11T20:18:44.228 INFO:scrape: 2024-04-11T20:18:44.228 INFO:scrape:Failure: "1712824918.6526558 mon.smithi089 (mon.0) 926 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log 2024-04-11T20:18:44.228 INFO:scrape:5 jobs: ['7651798', '7651679', '7651714', '7651762', '7651736'] 2024-04-11T20:18:44.228 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-04-11T20:18:44.228 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'roles', 'singlethreaded-fuse', 'supported-random-distros$/{centos_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy'] 2024-04-11T20:18:44.229 INFO:scrape: 2024-04-11T20:18:44.229 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-11T20:18:44.229 INFO:scrape:5 jobs: ['7651695', '7651870', '7651828', '7651742', '7651781'] 2024-04-11T20:18:44.229 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}'] 2024-04-11T20:18:44.229 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-11T20:18:44.229 INFO:scrape: 2024-04-11T20:18:44.229 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi032 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-11T20:18:44.230 INFO:scrape:2 jobs: ['7651815', '7651718'] 2024-04-11T20:18:44.230 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '3-modules}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'n/3', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'session_timeout}', 'standby-replay'] 2024-04-11T20:18:44.230 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '2-scrub/yes', '3-modules}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}', 'wsync/yes}'] 2024-04-11T20:18:44.230 INFO:scrape: 2024-04-11T20:18:44.230 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-04-11T20:18:44.230 INFO:scrape:1 jobs: ['7651786'] 2024-04-11T20:18:44.230 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-04-11T20:18:44.230 INFO:scrape: 2024-04-11T20:18:44.231 INFO:scrape:Failure: ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. 2024-04-11T20:18:44.231 INFO:scrape:2 jobs: ['7651750', '7651849'] 2024-04-11T20:18:44.231 INFO:scrape:suites intersection: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-pacific'] 2024-04-11T20:18:44.231 INFO:scrape:suites union: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-pacific'] 2024-04-11T20:18:44.231 INFO:scrape: 2024-04-11T20:18:44.231 INFO:scrape:Failure: Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-04-11T20:18:44.231 INFO:scrape:2 jobs: ['7651732', '7651741'] 2024-04-11T20:18:44.231 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-11T20:18:44.231 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{ignorelist_health', 'pg_health}', 'supported-random-distros$/{centos_latest}', 'tasks/mirror}', 'tasks/mirror}}'] 2024-04-11T20:18:44.231 INFO:scrape: 2024-04-11T20:18:44.232 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-11T20:18:44.232 INFO:scrape:3 jobs: ['7651791', '7651878', '7651810'] 2024-04-11T20:18:44.232 INFO:scrape:suites intersection: ['2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'pg_health', 'session_timeout}'] 2024-04-11T20:18:44.232 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'export-check', 'frag', 'fs/verify/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'ranks/5', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/dbench', 'tasks/{0-subvolume/{with-no-extra-options}', 'validater/lockdep}', 'wsync/no}', 'wsync/yes}'] 2024-04-11T20:18:44.232 INFO:scrape: 2024-04-11T20:18:44.232 INFO:scrape:Dead: b'2024-04-11T20:13:44.656 DEBUG:teuthology.exit:Finished running handlers' 2024-04-11T20:18:44.232 INFO:scrape:1 jobs: ['7651799'] 2024-04-11T20:18:44.232 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-04-11T20:18:44.232 INFO:scrape: 2024-04-11T20:18:44.233 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-04-11T20:18:44.233 INFO:scrape:1 jobs: ['7651686'] 2024-04-11T20:18:44.233 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-11T20:18:44.233 INFO:scrape: 2024-04-11T20:18:44.233 INFO:scrape:Failure: "2024-04-11T09:50:34.546875+0000 mds.a (mds.1) 26 : cluster [WRN] Scrub error on inode 0x20000007771 (/volumes/qa/sv_1/571d773f-17f4-4cb2-a046-4effed099cb9/client.0/tmp/t/linux-6.5.11/drivers/gpu/drm) see mds.a log and `damage ls` output for details" in cluster log 2024-04-11T20:18:44.233 INFO:scrape:1 jobs: ['7651861'] 2024-04-11T20:18:44.233 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-04-11T20:18:44.233 INFO:scrape: 2024-04-11T20:18:44.233 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-11T20:18:44.233 INFO:scrape:1 jobs: ['7651842'] 2024-04-11T20:18:44.234 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}'] 2024-04-11T20:18:44.234 INFO:scrape: 2024-04-11T20:18:44.234 INFO:scrape:Failure: Command failed on smithi107 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd' 2024-04-11T20:18:44.234 INFO:scrape:2 jobs: ['7651821', '7651776'] 2024-04-11T20:18:44.234 INFO:scrape:suites intersection: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install'] 2024-04-11T20:18:44.234 INFO:scrape:suites union: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/dbench}}', '3-workload/stress_tests/iozone}}', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install'] 2024-04-11T20:18:44.234 INFO:scrape: 2024-04-11T20:18:44.234 INFO:scrape:Failure: "2024-04-11T09:10:00.000206+0000 mon.smithi069 (mon.0) 524 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 31/219 objects degraded (14.155%), 12 pgs degraded" in cluster log 2024-04-11T20:18:44.234 INFO:scrape:1 jobs: ['7651817'] 2024-04-11T20:18:44.235 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-11T20:18:44.235 INFO:scrape: 2024-04-11T20:18:44.235 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-04-11T20:18:44.235 INFO:scrape:1 jobs: ['7651851'] 2024-04-11T20:18:44.235 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-11T20:18:44.235 INFO:scrape: 2024-04-11T20:18:44.235 INFO:scrape:Failure: "2024-04-11T08:25:52.131721+0000 mds.b (mds.0) 54 : cluster [WRN] Scrub error on inode 0x10000006794 (/volumes/qa/sv_1/68859f14-690c-4035-aed1-761562a0f919/client.0/tmp/t/linux-6.5.11/arch/x86/entry) see mds.b log and `damage ls` output for details" in cluster log 2024-04-11T20:18:44.235 INFO:scrape:1 jobs: ['7651775'] 2024-04-11T20:18:44.235 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-04-11T20:18:44.235 INFO:scrape: 2024-04-11T20:18:44.236 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-11T20:18:44.236 INFO:scrape:1 jobs: ['7651866'] 2024-04-11T20:18:44.236 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-04-11T20:18:44.236 INFO:scrape: 2024-04-11T20:18:44.236 INFO:scrape:Dead: None 2024-04-11T20:18:44.236 INFO:scrape:3 jobs: ['7651763', '7651802', '7651796'] 2024-04-11T20:18:44.236 INFO:scrape: 2024-04-11T20:18:44.249 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/ failed: 34 dead: 4 running: 0 waiting: 0 queued: 0 passed: 167 Fail ================================================================= [7651679] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651679/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651679/ "1712819853.9863389 mon.smithi088 (mon.0) 918 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7651686] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:16:19 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651686/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651686/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7651695] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:10 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651695/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651695/ reached maximum tries (51) after waiting for 300 seconds [7651714] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:39 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651714/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651714/ "2024-04-11T07:50:00.000255+0000 mon.smithi049 (mon.0) 614 : cluster [ERR] fs cephfs is degraded" in cluster log [7651718] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:26:20 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651718/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0d9d47b10f24b38ac0c83a794625628 Command failed (workunit test suites/fsx.sh) on smithi042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7651732] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/mirror} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651732/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c00f2aa87f2d490eb7a0906552f463c5 Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7651736] fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:43:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651736/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651736/ "2024-04-11T07:55:49.907152+0000 mon.a (mon.0) 321 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7651741] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror}} ----------------------------------------------------------------- time: 00:31:16 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651741/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46639d8efce34184a3d043e907af8d0f Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7651742] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651742/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651742/ reached maximum tries (51) after waiting for 300 seconds [7651750] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:11:35 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651750/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7eeed1cf59c644c383d2d43f87eb84c0 ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. [7651762] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:16 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651762/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651762/ "1712823078.7913888 mon.smithi088 (mon.0) 930 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7651769] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:21:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651769/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651769/ "2024-04-11T08:53:13.825239+0000 mds.f (mds.0) 1 : cluster [WRN] client.4663 isn't responding to mclientcaps(revoke), ino 0x100000030b0 pending pAsLsXsFcb issued pAsxLsXsxFscb, sent 301.564764 seconds ago" in cluster log [7651775] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:52:46 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651775/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651775/ "2024-04-11T08:25:52.131721+0000 mds.b (mds.0) 54 : cluster [WRN] Scrub error on inode 0x10000006794 (/volumes/qa/sv_1/68859f14-690c-4035-aed1-7615 62a0f919/client.0/tmp/t/linux-6.5.11/arch/x86/entry) see mds.b log and `damage ls` output for details" in cluster log [7651776] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} ----------------------------------------------------------------- time: 00:13:30 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651776/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04c970a654e64bf9b6ef809d269d9394 Command failed on smithi181 with status 1: 'sudo yum -y upgrade ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd' [7651780] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:15:36 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651780/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651780/ "2024-04-11T08:30:00.000229+0000 mon.a (mon.0) 1171 : cluster [WRN] pg 4.a is stuck peering for 62s, current state peering, last acting [1,10,6,7]" in cluster log [7651781] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:51 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651781/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651781/ reached maximum tries (51) after waiting for 300 seconds [7651786] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 07:31:38 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651786/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=760b573bfdd2403795fd14cdcced5f96 Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7651791] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:46:12 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651791/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=066ee73967da499aaba780c090d1030b Command failed (workunit test suites/dbench.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7651797] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651797/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27973d8b5618440eaf709fe575f2039d failed to install new kernel version within timeout [7651798] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:45 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651798/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651798/ "1712824918.6526558 mon.smithi089 (mon.0) 926 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7651810] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:32:15 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651810/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94014ed8ca0b4a74a4d5199949fb1daa Command failed (workunit test suites/dbench.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7651815] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:36:26 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651815/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54da02c693cc4332bbf833abd1f1b491 Command failed (workunit test suites/fsx.sh) on smithi032 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7651817] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651817/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651817/ "2024-04-11T09:10:00.000206+0000 mon.smithi069 (mon.0) 524 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 31/219 objects degraded (14.155%), 12 pgs degraded" in cluster log [7651821] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651821/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f92f671afe3b4afaba738c08c6832aae Command failed on smithi107 with status 1: 'sudo yum -y upgrade ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd' [7651828] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:01:40 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651828/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651828/ reached maximum tries (51) after waiting for 300 seconds [7651837] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/{ignorelist_health pg_health}} ----------------------------------------------------------------- time: 00:26:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651837/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651837/ SELinux denials found on ubuntu@smithi139.front.sepia.ceph.com: ['type=AVC msg=audit(1712825603.704:207): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7651842] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:28 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651842/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfe9e354adc44225baacfcff87c91445 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7651849] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:11:45 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651849/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e92ae163e3554531bbc0092026c55ab9 ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. [7651851] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:09:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651851/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651851/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7651859] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:23:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651859/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651859/ "2024-04-11T09:35:10.290845+0000 mds.b (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7651861] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:12:53 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651861/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651861/ "2024-04-11T09:50:34.546875+0000 mds.a (mds.1) 26 : cluster [WRN] Scrub error on inode 0x20000007771 (/volumes/qa/sv_1/571d773f-17f4-4cb2-a046-4eff ed099cb9/client.0/tmp/t/linux-6.5.11/drivers/gpu/drm) see mds.a log and `damage ls` output for details" in cluster log [7651866] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:38:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651866/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651866/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [7651870] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:59:49 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651870/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651870/ reached maximum tries (51) after waiting for 300 seconds [7651878] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:42:53 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651878/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7060b8f3a07462cb2a4c67c43daa501 Command failed (workunit test suites/dbench.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=71a5ccfeb986b7fb105908d4cabe37c4379637e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' Dead ================================================================= [7651763] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651763/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651763/ Error reimaging machines: Failed to power on smithi195 [7651796] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651796/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651796/ Error reimaging machines: list index out of range [7651799] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651799/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651799/ hit max job timeout [7651802] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651802/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651802/ Error reimaging machines: list index out of range Pass ================================================================= [7651677] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} time: 04:39:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651677/ [7651678] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:21:42 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651678/ [7651680] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651680/ [7651681] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs_python} time: 00:15:26 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651681/ [7651682] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:17:22 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651682/ [7651683] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:23:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651683/ [7651684] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:14:53 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651684/ [7651685] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:42:39 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651685/ [7651687] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:10:11 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651687/ [7651688] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:21:32 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651688/ [7651689] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 01:02:25 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651689/ [7651690] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:04:29 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651690/ [7651691] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:27:05 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651691/ [7651692] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:41:24 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651692/ [7651693] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:17:13 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651693/ [7651694] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:14:56 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651694/ [7651696] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:15:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651696/ [7651697] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:14:13 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651697/ [7651698] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:58:37 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651698/ [7651699] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:15:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651699/ [7651700] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 01:46:10 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651700/ [7651701] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:26:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651701/ [7651702] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:18 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651702/ [7651703] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:20:45 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651703/ [7651704] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:46:06 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651704/ [7651705] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:25:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651705/ [7651706] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:11 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651706/ [7651707] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:20:15 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651707/ [7651708] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} time: 00:40:09 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651708/ [7651709] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:13:54 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651709/ [7651710] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:28:43 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651710/ [7651711] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:28:23 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651711/ [7651712] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:46:37 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651712/ [7651713] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:39:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651713/ [7651715] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:11:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651715/ [7651716] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:21:17 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651716/ [7651717] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:10:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651717/ [7651719] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} time: 00:47:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651719/ [7651720] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:15:52 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651720/ [7651721] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:44:29 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651721/ [7651722] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:40:12 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651722/ [7651723] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:17:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651723/ [7651724] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:17:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651724/ [7651725] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:46 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651725/ [7651726] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} time: 00:10:15 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651726/ [7651727] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:37 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651727/ [7651728] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest clusters/small-cluster conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/trim-i24137} time: 00:12:59 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651728/ [7651729] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host overrides/{ignorelist_health pg_health}} time: 00:51:02 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651729/ [7651730] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-node-1-mds-1-osd conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mgr-osd-full} time: 00:20:28 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651730/ [7651731] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client} time: 00:13:02 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651731/ [7651733] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:25:02 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651733/ [7651734] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:09:09 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651734/ [7651735] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:39:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651735/ [7651737] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:16:49 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651737/ [7651738] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} time: 00:49:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651738/ [7651739] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:53:09 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651739/ [7651740] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:15:25 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651740/ [7651743] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:28:45 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651743/ [7651744] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:16 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651744/ [7651745] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:57:10 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651745/ [7651746] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:43 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651746/ [7651747] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/sessionmap} time: 00:19:56 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651747/ [7651748] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651748/ [7651749] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:24:45 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651749/ [7651751] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-iozone}} time: 02:17:31 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651751/ [7651752] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} time: 00:21:32 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651752/ [7651753] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} time: 02:04:58 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651753/ [7651754] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:15:29 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651754/ [7651755] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:38:44 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651755/ [7651756] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:54:34 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651756/ [7651757] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:14 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651757/ [7651758] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:13:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651758/ [7651759] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:15:25 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651759/ [7651760] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:24:25 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651760/ [7651761] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:34:27 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651761/ [7651764] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:38 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651764/ [7651765] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:21:32 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651765/ [7651766] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:17:51 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651766/ [7651767] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:27:19 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651767/ [7651768] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}} time: 00:17:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651768/ [7651770] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:17:07 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651770/ [7651771] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:08:13 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651771/ [7651772] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:43:48 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651772/ [7651773] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} time: 00:21:19 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651773/ [7651774] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:20:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651774/ [7651777] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/truncate_delay} time: 00:14:18 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651777/ [7651778] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:40:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651778/ [7651779] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/snapshot}} time: 00:38:29 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651779/ [7651782] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/dir-max-entries} time: 00:11:46 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651782/ [7651783] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:18:25 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651783/ [7651784] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:37:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651784/ [7651785] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651785/ [7651787] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:28 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651787/ [7651788] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/ino_release_cb} time: 00:13:53 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651788/ [7651789] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:17:07 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651789/ [7651790] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:20:43 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651790/ [7651792] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:14:53 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651792/ [7651793] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:09 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651793/ [7651794] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:08 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651794/ [7651795] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 02:02:26 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651795/ [7651800] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:14:14 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651800/ [7651801] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:38:48 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651801/ [7651803] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:28:43 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651803/ [7651804] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:33:11 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651804/ [7651805] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} time: 01:43:13 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651805/ [7651806] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:14:30 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651806/ [7651807] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:27 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651807/ [7651808] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:25:49 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651808/ [7651809] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:27 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651809/ [7651811] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:14:30 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651811/ [7651812] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:17:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651812/ [7651813] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:28 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651813/ [7651814] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:12:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651814/ [7651816] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:14:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651816/ [7651818] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:43 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651818/ [7651819] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:24:33 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651819/ [7651820] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:12:55 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651820/ [7651822] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:26 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651822/ [7651823] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:40:30 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651823/ [7651824] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:09:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651824/ [7651825] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:42:27 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651825/ [7651826] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:51:11 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651826/ [7651827] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-limits} time: 00:22:51 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651827/ [7651829] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:21:07 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651829/ [7651830] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:15:40 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651830/ [7651831] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:10:06 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651831/ [7651832] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:53:49 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651832/ [7651833] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:14:01 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651833/ [7651834] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:32:18 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651834/ [7651835] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:37:17 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651835/ [7651836] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:18:08 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651836/ [7651838] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}} time: 00:25:12 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651838/ [7651839] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:09:03 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651839/ [7651840] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:16:47 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651840/ [7651841] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:12 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651841/ [7651843] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 01:10:36 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651843/ [7651844] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-recovery} time: 00:48:56 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651844/ [7651845] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:22 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651845/ [7651846] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:41:00 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651846/ [7651847] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:25:16 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651847/ [7651848] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:24:56 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651848/ [7651850] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:47:17 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651850/ [7651852] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:46:35 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651852/ [7651853] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 00:54:31 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651853/ [7651854] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:22:24 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651854/ [7651855] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:11:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651855/ [7651856] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/exports} time: 00:44:24 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651856/ [7651857] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:50 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651857/ [7651858] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:23 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651858/ [7651860] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:12:58 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651860/ [7651862] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651862/ [7651863] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:19:12 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651863/ [7651864] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/fragment} time: 00:21:46 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651864/ [7651865] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:18 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651865/ [7651867] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:49:26 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651867/ [7651868] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:08:34 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651868/ [7651869] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:17:39 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651869/ [7651871] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:14:24 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651871/ [7651872] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:21:51 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651872/ [7651873] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:30:37 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651873/ [7651874] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:18:08 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651874/ [7651875] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:40:04 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651875/ [7651876] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds-flush} time: 00:12:27 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651876/ [7651877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:14:30 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651877/ [7651879] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds-full} time: 00:25:37 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651879/ [7651880] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:44 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651880/ [7651881] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} time: 00:12:41 info: https://pulpito.ceph.com/vshankar-2024-04-11_06:31:11-fs-wip-64730-2-testing-default-smithi/7651881/