2023-11-07T19:32:26.537 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-07T19:32:26.538 INFO:teuthology.results:subset: '859/1024'
2023-11-07T19:32:26.538 INFO:teuthology.results:seed: '9066'
2023-11-08T14:11:25.438 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-08T14:11:25.461 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-08T21:07:05.190 INFO:teuthology.results:Tests finished! gathering results...
2023-11-08T21:07:25.177 INFO:scrape:Found 203 jobs
2023-11-08T21:07:25.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450409/teuthology.log
2023-11-08T21:07:33.455 WARNING:scrape:Misunderstood line: 
2023-11-08T21:07:34.199 WARNING:scrape:Misunderstood line: 
2023-11-08T21:07:38.255 WARNING:scrape:Misunderstood line: 
2023-11-08T21:07:38.983 WARNING:scrape:Misunderstood line: 
2023-11-08T21:07:52.397 INFO:scrape:Found 16 distinct failure reasons
2023-11-08T21:07:52.398 INFO:scrape:Failure: Command failed on smithi194 with status 1: 'sudo yum -y install cephfs-java'
2023-11-08T21:07:52.399 INFO:scrape:2 jobs: ['7450448', '7450445']
2023-11-08T21:07:52.399 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-11-08T21:07:52.399 INFO:scrape:suites union: ['1-ranks/1', '1-upgrade}}', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-pacific']
2023-11-08T21:07:52.400 INFO:scrape:
2023-11-08T21:07:52.400 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi086 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-11-08T21:07:52.401 INFO:scrape:3 jobs: ['7450515', '7450557', '7450404']
2023-11-08T21:07:52.401 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'frag', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{client-shutdown', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-11-08T21:07:52.402 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-11-08T21:07:52.402 INFO:scrape:
2023-11-08T21:07:52.402 INFO:scrape:Failure: Command failed on smithi062 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7193-g96f02621-1jammy ceph-mds=18.0.0-7193-g96f02621-1jammy ceph-common=18.0.0-7193-g96f02621-1jammy ceph-fuse=18.0.0-7193-g96f02621-1jammy ceph-test=18.0.0-7193-g96f02621-1jammy radosgw=18.0.0-7193-g96f02621-1jammy python-ceph=18.0.0-7193-g96f02621-1jammy libcephfs1=18.0.0-7193-g96f02621-1jammy libcephfs-java=18.0.0-7193-g96f02621-1jammy libcephfs-jni=18.0.0-7193-g96f02621-1jammy librados2=18.0.0-7193-g96f02621-1jammy librbd1=18.0.0-7193-g96f02621-1jammy rbd-fuse=18.0.0-7193-g96f02621-1jammy python3-cephfs=18.0.0-7193-g96f02621-1jammy cephfs-shell=18.0.0-7193-g96f02621-1jammy cephfs-top=18.0.0-7193-g96f02621-1jammy cephfs-mirror=18.0.0-7193-g96f02621-1jammy'
2023-11-08T21:07:52.403 INFO:scrape:1 jobs: ['7450449']
2023-11-08T21:07:52.403 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/truncate_delay}']
2023-11-08T21:07:52.403 INFO:scrape:
2023-11-08T21:07:52.404 INFO:scrape:Dead: None
2023-11-08T21:07:52.404 INFO:scrape:11 jobs: ['7450409', '7450408', '7450403', '7450402', '7450392', '7450391', '7450382', '7450380', '7450432', '7450416', '7450418']
2023-11-08T21:07:52.405 INFO:scrape:
2023-11-08T21:07:52.405 INFO:scrape:Failure: Command failed on smithi124 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b615f8-7e3a-11ee-8dbb-212e2dc638e7 -- ceph mon dump -f json'
2023-11-08T21:07:52.406 INFO:scrape:3 jobs: ['7450514', '7450536', '7450495']
2023-11-08T21:07:52.406 INFO:scrape:suites intersection: []
2023-11-08T21:07:52.406 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'supported-random-distros$/{centos_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy']
2023-11-08T21:07:52.407 INFO:scrape:
2023-11-08T21:07:52.407 INFO:scrape:Failure: Stale jobs detected, aborting.
2023-11-08T21:07:52.407 INFO:scrape:1 jobs: ['7450423']
2023-11-08T21:07:52.408 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snap-schedule}']
2023-11-08T21:07:52.409 INFO:scrape:
2023-11-08T21:07:52.409 INFO:scrape:Failure: Test failure: test_idem_unaffected_root_squash (tasks.cephfs.test_admin.TestFsAuthorizeUpdate)
2023-11-08T21:07:52.410 INFO:scrape:1 jobs: ['7450470']
2023-11-08T21:07:52.410 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/admin}']
2023-11-08T21:07:52.410 INFO:scrape:
2023-11-08T21:07:52.411 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi040 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-11-08T21:07:52.411 INFO:scrape:2 jobs: ['7450508', '7450439']
2023-11-08T21:07:52.412 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-11-08T21:07:52.412 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'ubuntu_latest}}']
2023-11-08T21:07:52.412 INFO:scrape:
2023-11-08T21:07:52.413 INFO:scrape:Failure: Command failed on smithi077 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib'
2023-11-08T21:07:52.413 INFO:scrape:3 jobs: ['7450450', '7450453', '7450444']
2023-11-08T21:07:52.413 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-11-08T21:07:52.414 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-mount/mount/fuse', '1-tests/iozone}}}', '2-logrotate}', '2-scrub/yes', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/ffsb}}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/fsync-tester}}', 'begin/{0-install', 'centos_8.stream', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/osd-mds-delay', 'multifs', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-from/nautilus', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'wsync/no}']
2023-11-08T21:07:52.414 INFO:scrape:
2023-11-08T21:07:52.415 INFO:scrape:Failure: ['Connection timed out', 'Failed to connect to the host via ssh: ssh: connect to host smithi055.front.sepia.ceph.com port 22: No route to host', 'See stdout/stderr for the exact error', 'The module failed to execute correctly, you probably need to set the interpreter.', 'ubuntu@smithi055.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).']
2023-11-08T21:07:52.415 INFO:scrape:1 jobs: ['7450421']
2023-11-08T21:07:52.415 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2023-11-08T21:07:52.416 INFO:scrape:
2023-11-08T21:07:52.416 INFO:scrape:Failure: No module named 'tasks.ceph'
2023-11-08T21:07:52.417 INFO:scrape:3 jobs: ['7450447', '7450446', '7450452']
2023-11-08T21:07:52.417 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-11-08T21:07:52.417 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/test_journal_migration}', 'tasks/workunit/dir-max-entries}', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-11-08T21:07:52.418 INFO:scrape:
2023-11-08T21:07:52.418 INFO:scrape:Valgrind: client (Leak_StillReachable), mon (Leak_StillReachable)
2023-11-08T21:07:52.419 INFO:scrape:2 jobs: ['7450511', '7450428']
2023-11-08T21:07:52.419 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-11-08T21:07:52.419 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}']
2023-11-08T21:07:52.420 INFO:scrape:
2023-11-08T21:07:52.421 INFO:scrape:Valgrind: cephfs-mirror-client (Leak_StillReachable)
2023-11-08T21:07:52.421 INFO:scrape:1 jobs: ['7450541']
2023-11-08T21:07:52.422 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}']
2023-11-08T21:07:52.423 INFO:scrape:
2023-11-08T21:07:52.424 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-11-08T21:07:52.425 INFO:scrape:1 jobs: ['7450502']
2023-11-08T21:07:52.425 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}']
2023-11-08T21:07:52.426 INFO:scrape:
2023-11-08T21:07:52.426 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-11-08T21:07:52.427 INFO:scrape:1 jobs: ['7450455']
2023-11-08T21:07:52.427 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-11-08T21:07:52.427 INFO:scrape:
2023-11-08T21:07:52.428 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2023-11-08T21:07:52.428 INFO:scrape:1 jobs: ['7450398']
2023-11-08T21:07:52.428 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2023-11-08T21:07:52.429 INFO:scrape:
2023-11-08T21:07:52.439 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/
failed:  26
dead:    11
running: 0
waiting: 0
queued:  0
passed:  166


Fail
=================================================================
[7450398]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:49:29
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450398/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450398/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1ccbfc1319f475ba2b751cf1c9cb739

    Command failed (workunit test suites/ffsb.sh) on smithi005 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7450404]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:47:42
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450404/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450404/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6396806d96d04694a01634838eb30637

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi096
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7450421]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:11:17
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450421/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450421/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2383fd5d4104fd79ab70ff8c70c2369

    ['Connection timed out', 'Failed to connect to the host via ssh: ssh:
    connect to host smithi055.front.sepia.ceph.com port 22: No route to host',
    'See stdout/stderr for the exact error', 'The module failed to execute
    correctly, you probably need to set the interpreter.',
    'ubuntu@smithi055.front.sepia.ceph.com: Permission denied
    (publickey,password,keyboard-interactive).']

[7450423]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450423/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450423/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cc13e70dd634720ae8e03af40dcb6bb

    Stale jobs detected, aborting.

[7450428]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:52:50
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450428/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450428/

    saw valgrind issues

[7450439]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:26:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450439/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450439/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ec7e817ac1b449ba1217bb6a121302e

    Command failed (workunit test kernel_untar_build.sh) on smithi153 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7450444]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:13:52
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450444/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450444/
sentry: https://sentry.ceph.com/organizations/ceph/?query=138d528012d5415692d4c956d8a5e8e5

    Command failed on smithi161 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1
    librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-
    mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib'

[7450445]  fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
-----------------------------------------------------------------
time:   00:13:03
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450445/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450445/
sentry: https://sentry.ceph.com/organizations/ceph/?query=860874f5cb064524b0ca9ae0df4492f1

    Command failed on smithi039 with status 1: 'sudo yum -y install cephfs-
    java'

[7450446]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:12:03
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450446/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450446/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60e41edfb1314dd2a47ec0a42b2fad7f

    No module named 'tasks.ceph'

[7450447]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:11:21
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450447/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450447/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d3fac94f9294786ac761bcdf9289f0a

    No module named 'tasks.ceph'

[7450448]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:13:04
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450448/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450448/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dd26c378206481c87ccea5c41af9ff9

    Command failed on smithi194 with status 1: 'sudo yum -y install cephfs-
    java'

[7450449]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
-----------------------------------------------------------------
time:   00:09:03
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450449/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450449/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab873f93302748109ee5357efeb4eaea

    Command failed on smithi062 with status 100: 'sudo
    DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="
    --force-confdef" -o Dpkg::Options::="--force-confold" install
    ceph=18.0.0-7193-g96f02621-1jammy ceph-mds=18.0.0-7193-g96f02621-1jammy
    ceph-common=18.0.0-7193-g96f02621-1jammy ceph-
    fuse=18.0.0-7193-g96f02621-1jammy ceph-test=18.0.0-7193-g96f02621-1jammy
    radosgw=18.0.0-7193-g96f02621-1jammy python-
    ceph=18.0.0-7193-g96f02621-1jammy libcephfs1=18.0.0-7193-g96f02621-1jammy
    libcephfs-java=18.0.0-7193-g96f02621-1jammy libcephfs-
    jni=18.0.0-7193-g96f02621-1jammy librados2=18.0.0-7193-g96f02621-1jammy
    librbd1=18.0.0-7193-g96f02621-1jammy rbd-fuse=18.0.0-7193-g96f02621-1jammy
    python3-cephfs=18.0.0-7193-g96f02621-1jammy cephfs-
    shell=18.0.0-7193-g96f02621-1jammy cephfs-top=18.0.0-7193-g96f02621-1jammy
    cephfs-mirror=18.0.0-7193-g96f02621-1jammy'

[7450450]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:10:28
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450450/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450450/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7247b030a4a240ccb9ba7774724d5af8

    Command failed on smithi077 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1
    librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-
    mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib'

[7450452]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
-----------------------------------------------------------------
time:   00:11:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450452/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450452/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8c73c001e0ef43ae896b97d4d34276e8

    No module named 'tasks.ceph'

[7450453]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
-----------------------------------------------------------------
time:   00:11:14
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450453/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e5784e3660ed4c5fbd29eacc18ce0384

    Command failed on smithi096 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1
    librados2 librbd1 python-ceph rbd-fuse python-rados python-rgw python-rbd
    python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-
    devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux'

[7450455]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   08:46:43
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450455/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450455/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff546086a520470a89f0f5d98955d307

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7450470]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:44:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450470/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450470/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8cf92ab8606b407eba87424374387868

    Test failure: test_idem_unaffected_root_squash
    (tasks.cephfs.test_admin.TestFsAuthorizeUpdate)

[7450495]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:24
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450495/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450495/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dae8ad11a91f4cf2b3b2cb954adee5fb

    Command failed on smithi079 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    eece31f2-7e37-11ee-8dbb-212e2dc638e7 -- ceph mon dump -f json'

[7450502]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   02:11:32
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450502/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450502/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7450508]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:32:08
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450508/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450508/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ace6851d5a1c48e09d601cd65df602c5

    Command failed (workunit test kernel_untar_build.sh) on smithi040 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7450511]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:48:57
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450511/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450511/

    saw valgrind issues

[7450514]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:22
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450514/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450514/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c3d92b3a26446efab94535c1041a4d9

    Command failed on smithi124 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    79b615f8-7e3a-11ee-8dbb-212e2dc638e7 -- ceph mon dump -f json'

[7450515]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:21:24
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450515/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450515/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20d8e9efab1c4f869e2d1394dc10037d

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi086
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7450536]  fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:13:01
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450536/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450536/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f877358bc75b43688db58e5d607c3e2b

    Command failed on smithi073 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:96f02621f272b8c7e4497bd706bb39e99349a1d4 pull'

[7450541]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:27:55
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450541/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450541/

    saw valgrind issues

[7450557]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:45:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450557/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450557/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f26564384c36414f949f7f3b5934fc80

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi042
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=96f02621f272b8c7e4497bd706bb39e99349a1d4
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'



Dead
=================================================================
[7450380]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450380/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450380/

    Error reimaging machines: Expected smithi055's OS to be rhel 8.6 but found
    ubuntu 22.04

[7450382]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450382/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450382/

    Error reimaging machines: list index out of range

[7450391]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450391/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450391/

    Error reimaging machines: Expected smithi055's OS to be centos 9 but found
    ubuntu 22.04

[7450392]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450392/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450392/

    Error reimaging machines: list index out of range

[7450402]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450402/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450402/

    Error reimaging machines: Expected smithi055's OS to be ubuntu 22.04 but
    found centos 9

[7450403]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450403/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450403/

    Error reimaging machines: list index out of range

[7450408]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450408/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450408/

    Error reimaging machines: Expected smithi055's OS to be ubuntu 22.04 but
    found centos 9

[7450409]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450409/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450409/

    Error reimaging machines: list index out of range

[7450416]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450416/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450416/

    Error reimaging machines: Expected smithi055's OS to be ubuntu 22.04 but
    found rhel 8.6

[7450418]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450418/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450418/

    Error reimaging machines: list index out of range

[7450432]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450432/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450432/

    Error reimaging machines: list index out of range



Pass
=================================================================
[7450374] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:22:55
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450374/

[7450375] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:24:59
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450375/

[7450376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:15:23
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450376/

[7450377] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:12
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450377/

[7450378] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
time:   00:44:55
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450378/

[7450379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:26:57
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450379/

[7450381] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:16:29
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450381/

[7450383] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:15:02
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450383/

[7450384] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:20:16
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450384/

[7450385] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:35
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450385/

[7450386] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:21:34
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450386/

[7450387] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}}
time:   02:05:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450387/

[7450388] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:39
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450388/

[7450389] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
time:   01:04:30
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450389/

[7450390] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:36:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450390/

[7450393] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:24
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450393/

[7450394] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep}
time:   00:20:36
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450394/

[7450395] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   00:52:11
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450395/

[7450396] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:17:23
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450396/

[7450397] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:36:28
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450397/

[7450399] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:43:12
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450399/

[7450400] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:23:40
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450400/

[7450401] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:17:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450401/

[7450405] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:49
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450405/

[7450406] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:16:27
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450406/

[7450407] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:33
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450407/

[7450410] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
time:   00:28:08
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450410/

[7450411] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:11
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450411/

[7450412] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:42
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450412/

[7450413] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:14:25
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450413/

[7450414] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:21:11
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450414/

[7450415] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:23:27
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450415/

[7450417] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:23:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450417/

[7450419] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:43
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450419/

[7450420] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:20:23
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450420/

[7450422] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:13
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450422/

[7450424] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   00:54:35
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450424/

[7450425] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:19:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450425/

[7450426] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:15:57
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450426/

[7450427] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
time:   00:32:09
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450427/

[7450429] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:23
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450429/

[7450430] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:24:51
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450430/

[7450431] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:25:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450431/

[7450433] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:40:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450433/

[7450434] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450434/

[7450435] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:27:54
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450435/

[7450436] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:39:39
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450436/

[7450437] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:20:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450437/

[7450438] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:04
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450438/

[7450440] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:51:47
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450440/

[7450441] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:29
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450441/

[7450442] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:18:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450442/

[7450443] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:36:23
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450443/

[7450451] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:35:48
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450451/

[7450454] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:44:41
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450454/

[7450456] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:33
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450456/

[7450457] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:32
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450457/

[7450458] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450458/

[7450459] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
time:   00:45:56
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450459/

[7450460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:17:50
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450460/

[7450461] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:18:40
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450461/

[7450462] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:44
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450462/

[7450463] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:20:43
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450463/

[7450464] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}}
time:   02:27:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450464/

[7450465] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:15:42
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450465/

[7450466] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:17:12
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450466/

[7450467] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:36:42
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450467/

[7450468] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:16:48
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450468/

[7450469] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:47:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450469/

[7450471] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:25:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450471/

[7450472] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep}
time:   00:19:59
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450472/

[7450473] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
time:   00:27:21
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450473/

[7450474] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:19:27
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450474/

[7450475] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:43:16
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450475/

[7450476] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:07:50
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450476/

[7450477] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:15:24
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450477/

[7450478] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:37:33
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450478/

[7450479] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:19:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450479/

[7450480] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:27
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450480/

[7450481] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
time:   00:40:24
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450481/

[7450482] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:15
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450482/

[7450483] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:16:11
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450483/

[7450484] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:51
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450484/

[7450485] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:16:59
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450485/

[7450486] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:27:17
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450486/

[7450487] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:15:46
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450487/

[7450488] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:37
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450488/

[7450489] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:21:32
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450489/

[7450490] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:15:40
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450490/

[7450491] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:07:57
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450491/

[7450492] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
time:   00:26:52
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450492/

[7450493] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:15:10
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450493/

[7450494] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:21:16
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450494/

[7450496] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:22:58
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450496/

[7450497] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:36:07
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450497/

[7450498] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:27:15
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450498/

[7450499] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:14:56
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450499/

[7450500] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:51
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450500/

[7450501] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:15:54
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450501/

[7450503] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:20:48
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450503/

[7450504] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:09
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450504/

[7450505] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   01:03:28
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450505/

[7450506] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:31
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450506/

[7450507] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450507/

[7450509] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:17:39
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450509/

[7450510] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:23:37
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450510/

[7450512] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:26:15
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450512/

[7450513] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
time:   00:44:19
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450513/

[7450516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:49:56
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450516/

[7450517] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:48
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450517/

[7450518] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   01:05:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450518/

[7450519] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:25:56
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450519/

[7450520] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450520/

[7450521] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}}
time:   00:48:59
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450521/

[7450522] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:18:14
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450522/

[7450523] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:41:08
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450523/

[7450524] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:21:52
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450524/

[7450525] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:38
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450525/

[7450526] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:24:03
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450526/

[7450527] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:33
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450527/

[7450528] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:16:21
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450528/

[7450529] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   01:00:06
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450529/

[7450530] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:24:30
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450530/

[7450531] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:15:32
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450531/

[7450532] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror}
time:   02:34:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450532/

[7450533] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   00:25:13
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450533/

[7450534] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:00:52
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450534/

[7450535] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:21:53
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450535/

[7450537] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:18:31
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450537/

[7450538] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:50:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450538/

[7450539] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:02:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450539/

[7450540] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:16:30
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450540/

[7450542] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:24:35
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450542/

[7450543] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:38:25
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450543/

[7450544] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:25
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450544/

[7450545] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:18:45
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450545/

[7450546] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:15:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450546/

[7450547] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:20
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450547/

[7450548] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:15:52
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450548/

[7450549] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:26:42
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450549/

[7450550] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:18:25
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450550/

[7450551] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}}
time:   02:27:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450551/

[7450552] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:22:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450552/

[7450553] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
time:   01:05:22
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450553/

[7450554] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:29:36
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450554/

[7450555] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:24:00
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450555/

[7450556] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:58
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450556/

[7450558] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
time:   00:19:04
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450558/

[7450559] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:32:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450559/

[7450560] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:51:47
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450560/

[7450561] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:03
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450561/

[7450562] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:43:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450562/

[7450563] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:28
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450563/

[7450564] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:14
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450564/

[7450565] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:23:27
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450565/

[7450566] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:34:04
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450566/

[7450567] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:18:48
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450567/

[7450568] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:17:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450568/

[7450569] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:19:38
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450569/

[7450570] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:16:43
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450570/

[7450571] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:28:06
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450571/

[7450572] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:48:58
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450572/

[7450573] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:24:18
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450573/

[7450574] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:17:11
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450574/

[7450575] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}}
time:   00:41:05
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450575/

[7450576] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:13:26
info:   https://pulpito.ceph.com/vshankar-2023-11-07_06:04:04-fs-wip-vshankar-testing-20231107.042705-testing-default-smithi/7450576/