2023-05-03T15:17:08.491 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-03T15:17:08.492 INFO:teuthology.results:subset: '630/1024'
2023-05-03T15:17:08.492 INFO:teuthology.results:seed: '3879'
2023-05-03T21:08:41.523 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-03T21:08:41.585 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-04T08:45:42.511 INFO:teuthology.results:Tests finished! gathering results...
2023-05-04T08:46:00.453 INFO:scrape:Found 203 jobs
2023-05-04T08:46:06.441 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261125/teuthology.log
2023-05-04T08:46:06.442 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261125/teuthology.log
2023-05-04T08:46:06.443 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261125/teuthology.log
2023-05-04T08:46:09.800 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261080/teuthology.log
2023-05-04T08:46:09.801 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261080/teuthology.log
2023-05-04T08:46:10.111 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261111/teuthology.log
2023-05-04T08:46:10.112 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261111/teuthology.log
2023-05-04T08:46:10.364 WARNING:scrape:Misunderstood line: 
2023-05-04T08:46:15.758 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261098/teuthology.log
2023-05-04T08:46:15.759 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261098/teuthology.log
2023-05-04T08:46:30.517 WARNING:scrape:Misunderstood line: 
2023-05-04T08:46:35.986 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261146/teuthology.log
2023-05-04T08:46:35.987 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261146/teuthology.log
2023-05-04T08:46:37.586 WARNING:scrape:Misunderstood line: 
2023-05-04T08:46:39.581 INFO:scrape:Found 23 distinct failure reasons
2023-05-04T08:46:39.582 INFO:scrape:Failure: Command failed on smithi081 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib'
2023-05-04T08:46:39.582 INFO:scrape:32 jobs
2023-05-04T08:46:39.583 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}']
2023-05-04T08:46:39.583 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', '2-workunit/suites/pjd}}', 'bluestore-bitmap', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1-mds-1-client', 'clusters/1-mds-1-client-coloc', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'supported-random-distros$/{centos_8}', 'tasks/alternate-pool}', 'tasks/asok_dump_tree}', 'tasks/backtrace}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-recovery}', 'tasks/exports}', 'tasks/fscrypt-iozone}', 'tasks/fscrypt-pjd}', 'tasks/fsstress', 'tasks/ino_release_cb}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/libcephfs/{frag', 'tasks/mgr-osd-full}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/openfiletable}', 'tasks/recovery-fs}', 'tasks/strays}', 'tasks/trim-i24137}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/clone}}', 'test}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}']
2023-05-04T08:46:39.584 INFO:scrape:
2023-05-04T08:46:39.584 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-05-04T08:46:39.584 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fcf5c75fce0]
 2: gsignal()
 3: abort()
 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fcf5bd7609b]
 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fcf5bd7c53c]
 6: /lib64/libstdc++.so.6(+0x96597) [0x7fcf5bd7c597]
 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fcf5bd7c7f8]
 8: /lib64/libstdc++.so.6(+0x9204b) [0x7fcf5bd7804b]
 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3cc) [0x55c5af59da8c]
 10: (MDSMonitor::tick()+0x18a) [0x55c5af5ac3fa]
 11: (Monitor::tick()+0xc0) [0x55c5af358b30]
 12: (Context::complete(int)+0xd) [0x55c5af37c0ad]
 13: (CommonSafeTimer<std::mutex>::timer_thread()+0x127) [0x7fcf5f036307]
 14: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7fcf5f037431]
 15: /lib64/libpthread.so.0(+0x81cf) [0x7fcf5c7551cf]
 16: clone()
2023-05-04T08:46:39.585 INFO:scrape:22 jobs: ['7261197', '7261203', '7261151', '7261154', '7261100', '7261020', '7261059', '7261103', '7261071', '7261049', '7261170', '7261085', '7261161', '7261075', '7261060', '7261089', '7261073', '7261094', '7261083', '7261007', '7261175', '7261014']
2023-05-04T08:46:39.585 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-05-04T08:46:39.586 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/admin}', 'tasks/auto-repair}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/damage}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/mds-flush}', 'tasks/mds-full}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/snap_schedule_snapdir}', 'tasks/snapshots}', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/basic}}', 'test/clone}}', 'test/misc}}', 'thrashosds-health}']
2023-05-04T08:46:39.586 INFO:scrape:
2023-05-04T08:46:39.586 INFO:scrape:Dead: b'2023-05-04T07:11:29.241 DEBUG:teuthology.exit:Finished running handlers'
2023-05-04T08:46:39.587 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f69093ff420]
 2: gsignal()
 3: abort()
 4: /usr/lib/ceph/libceph-common.so.2(+0x3692b7) [0x7f69089692b7]
 5: (__cxxabiv1::__terminate(void (*)())+0xc) [0x7f690901ea2c]
 6: /usr/lib/ceph/libceph-common.so.2(+0xa1ea97) [0x7f690901ea97]
 7: /usr/lib/ceph/libceph-common.so.2(+0xa1ebf9) [0x7f690901ebf9]
 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f690896ad42]
 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3bc) [0x56194fddcaac]
 10: (MDSMonitor::tick()+0x18a) [0x56194fdeb41a]
 11: (Monitor::tick()+0xc0) [0x56194fba2840]
 12: (Context::complete(int)+0xd) [0x56194fbc29dd]
 13: (CommonSafeTimer<std::mutex>::timer_thread()+0x12c) [0x7f6908a1006c]
 14: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f6908a11101]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f69093f3609]
 16: clone()
2023-05-04T08:46:39.587 INFO:scrape:1 jobs: ['7261142']
2023-05-04T08:46:39.587 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2023-05-04T08:46:39.588 INFO:scrape:
2023-05-04T08:46:39.588 INFO:scrape:Failure: Command failed on smithi047 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'
2023-05-04T08:46:39.588 INFO:scrape:6 jobs: ['7261178', '7261206', '7261155', '7261029', '7261058', '7261090']
2023-05-04T08:46:39.588 INFO:scrape:suites intersection: ['1-mount/mount/fuse', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}']
2023-05-04T08:46:39.589 INFO:scrape:suites union: ['1-client-sanity}}}', '1-mount/mount/fuse', '1-tests/blogbench}}}', '1-tests/dbench}}}', '1-tests/fsstress}}}', '1-tests/iozone}}}', '1-tests/kernel_untar_build}}}', '2-workload/new_ops/{0-clients/fuse-upgrade', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus', 'tasks/{0-from/pacific']
2023-05-04T08:46:39.589 INFO:scrape:
2023-05-04T08:46:39.589 INFO:scrape:Timeout 120 running ceph --cluster ceph quorum_status
2023-05-04T08:46:39.590 INFO:scrape:1 jobs: ['7261050']
2023-05-04T08:46:39.590 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-05-04T08:46:39.590 INFO:scrape:
2023-05-04T08:46:39.591 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-05-04T08:46:39.591 INFO:scrape:1 jobs: ['7261109']
2023-05-04T08:46:39.591 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2023-05-04T08:46:39.591 INFO:scrape:
2023-05-04T08:46:39.592 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'
2023-05-04T08:46:39.592 INFO:scrape:2 jobs: ['7261182', '7261012']
2023-05-04T08:46:39.592 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-05-04T08:46:39.593 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-05-04T08:46:39.593 INFO:scrape:
2023-05-04T08:46:39.593 INFO:scrape:Dead: b'2023-05-04T03:26:28.016 DEBUG:teuthology.exit:Finished running handlers'
2023-05-04T08:46:39.594 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fda65242520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2bbe) [0x7fda656a2bbe]
 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae24c) [0x7fda656ae24c]
 7: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae2b7) [0x7fda656ae2b7]
 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae518) [0x7fda656ae518]
 9: (std::__throw_out_of_range(char const*)+0x40) [0x7fda656a54e6]
 10: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3ac) [0x5586d2240b7c]
 11: (MDSMonitor::tick()+0x182) [0x5586d224e722]
 12: (Monitor::tick()+0xb0) [0x5586d20c4540]
 13: ceph-mon(+0xb0816d) [0x5586d206e16d]
 14: (CommonSafeTimer<std::mutex>::timer_thread()+0x116) [0x7fda66045726]
 15: /usr/lib/ceph/libceph-common.so.2(+0x245f71) [0x7fda66045f71]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fda65294b43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fda65326a00]
2023-05-04T08:46:39.594 INFO:scrape:1 jobs: ['7261009']
2023-05-04T08:46:39.594 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-05-04T08:46:39.594 INFO:scrape:
2023-05-04T08:46:39.595 INFO:scrape:Dead: b'2023-05-04T04:04:32.297 DEBUG:teuthology.exit:Finished running handlers'
2023-05-04T08:46:39.595 INFO:scrape:4 jobs: ['7261024', '7261177', '7261120', '7261138']
2023-05-04T08:46:39.595 INFO:scrape:suites intersection: ['2-logrotate}']
2023-05-04T08:46:39.596 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/ffsb}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'frag', 'fs/functional/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{rhel_8}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'workloads/cephfs-mirror-ha-workunit}', 'wsync/no}', 'wsync/yes}']
2023-05-04T08:46:39.596 INFO:scrape:
2023-05-04T08:46:39.596 INFO:scrape:Failure: Command failed on smithi141 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-05-04T08:46:39.596 INFO:scrape:3 jobs: ['7261207', '7261174', '7261070']
2023-05-04T08:46:39.597 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-05-04T08:46:39.597 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-05-04T08:46:39.597 INFO:scrape:
2023-05-04T08:46:39.598 INFO:scrape:Dead: None
2023-05-04T08:46:39.598 INFO:scrape:5 jobs: ['7261125', '7261080', '7261111', '7261098', '7261146']
2023-05-04T08:46:39.598 INFO:scrape:
2023-05-04T08:46:39.599 INFO:scrape:Crash: Command failed (workunit test suites/fsstress.sh) on smithi186 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2023-05-04T08:46:39.599 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1da0c42520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2bbe) [0x7f1da10a2bbe]
 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae24c) [0x7f1da10ae24c]
 7: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae2b7) [0x7f1da10ae2b7]
 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae518) [0x7f1da10ae518]
 9: (std::__throw_out_of_range(char const*)+0x40) [0x7f1da10a54e6]
 10: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3ac) [0x5652cd02cb7c]
 11: (MDSMonitor::tick()+0x182) [0x5652cd03a722]
 12: (Monitor::tick()+0xb0) [0x5652cceb0540]
 13: ceph-mon(+0xb0816d) [0x5652cce5a16d]
 14: (CommonSafeTimer<std::mutex>::timer_thread()+0x116) [0x7f1da1a45726]
 15: /usr/lib/ceph/libceph-common.so.2(+0x245f71) [0x7f1da1a45f71]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1da0c94b43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1da0d26a00]
2023-05-04T08:46:39.599 INFO:scrape:9 jobs: ['7261126', '7261045', '7261019', '7261102', '7261026', '7261139', '7261028', '7261108', '7261134']
2023-05-04T08:46:39.600 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-05-04T08:46:39.600 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/iozone}}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'session_timeout', 'tasks/asok_dump_tree}', 'tasks/cephfs_misc_tests}', 'tasks/fscrypt-common}', 'tasks/fscrypt-ffsb}', 'tasks/mdtest}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'test/snapshot}}', 'thrashosds-health}']
2023-05-04T08:46:39.600 INFO:scrape:
2023-05-04T08:46:39.600 INFO:scrape:Crash: Test failure: test_r_with_no_fsname_and_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
2023-05-04T08:46:39.601 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3a02fbd420]
 2: gsignal()
 3: abort()
 4: /usr/lib/ceph/libceph-common.so.2(+0x3692b7) [0x7f3a035692b7]
 5: (__cxxabiv1::__terminate(void (*)())+0xc) [0x7f3a03c1ea2c]
 6: /usr/lib/ceph/libceph-common.so.2(+0xa1ea97) [0x7f3a03c1ea97]
 7: /usr/lib/ceph/libceph-common.so.2(+0xa1ebf9) [0x7f3a03c1ebf9]
 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f3a0356ad42]
 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3bc) [0x557fbcc78aac]
 10: (MDSMonitor::tick()+0x18a) [0x557fbcc8741a]
 11: (Monitor::tick()+0xc0) [0x557fbca3e840]
 12: (Context::complete(int)+0xd) [0x557fbca5e9dd]
 13: (CommonSafeTimer<std::mutex>::timer_thread()+0x12c) [0x7f3a0361006c]
 14: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f3a03611101]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3a02fb1609]
 16: clone()
2023-05-04T08:46:39.601 INFO:scrape:4 jobs: ['7261208', '7261188', '7261128', '7261037']
2023-05-04T08:46:39.601 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_20.04}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-05-04T08:46:39.602 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/admin}', 'tasks/multifs-auth}', 'tasks/sessionmap}']
2023-05-04T08:46:39.602 INFO:scrape:
2023-05-04T08:46:39.602 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-05-04T08:46:39.603 INFO:scrape:1 jobs: ['7261119']
2023-05-04T08:46:39.603 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-05-04T08:46:39.603 INFO:scrape:
2023-05-04T08:46:39.603 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi182 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-05-04T08:46:39.604 INFO:scrape:1 jobs: ['7261087']
2023-05-04T08:46:39.604 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-05-04T08:46:39.604 INFO:scrape:
2023-05-04T08:46:39.605 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-05-04T08:46:39.605 INFO:scrape:3 jobs: ['7261164', '7261006', '7261081']
2023-05-04T08:46:39.605 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-05-04T08:46:39.606 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-05-04T08:46:39.606 INFO:scrape:
2023-05-04T08:46:39.606 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-05-04T08:46:39.606 INFO:scrape:1 jobs: ['7261076']
2023-05-04T08:46:39.607 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}']
2023-05-04T08:46:39.607 INFO:scrape:
2023-05-04T08:46:39.607 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-05-04T08:46:39.608 INFO:scrape:1 jobs: ['7261039']
2023-05-04T08:46:39.608 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}']
2023-05-04T08:46:39.608 INFO:scrape:
2023-05-04T08:46:39.608 INFO:scrape:Failure: Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan)
2023-05-04T08:46:39.609 INFO:scrape:1 jobs: ['7261065']
2023-05-04T08:46:39.609 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2023-05-04T08:46:39.609 INFO:scrape:
2023-05-04T08:46:39.610 INFO:scrape:Failure: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2023-05-04T08:46:39.610 INFO:scrape:1 jobs: ['7261079']
2023-05-04T08:46:39.610 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/fragment}']
2023-05-04T08:46:39.610 INFO:scrape:
2023-05-04T08:46:39.611 INFO:scrape:Failure: Test failure: test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc), test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc)
2023-05-04T08:46:39.611 INFO:scrape:1 jobs: ['7261036']
2023-05-04T08:46:39.611 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2023-05-04T08:46:39.612 INFO:scrape:
2023-05-04T08:46:39.612 INFO:scrape:Crash: Test failure: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules), test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2023-05-04T08:46:39.612 INFO:scrape:ceph version 18.0.0-3719-g221532d2 (221532d2ce1c467630b0dec52a6c3ecd8d441799) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb08f6b8420]
 2: gsignal()
 3: abort()
 4: /usr/lib/ceph/libceph-common.so.2(+0x3692b7) [0x7fb08eb692b7]
 5: (__cxxabiv1::__terminate(void (*)())+0xc) [0x7fb08f21ea2c]
 6: /usr/lib/ceph/libceph-common.so.2(+0xa1ea97) [0x7fb08f21ea97]
 7: /usr/lib/ceph/libceph-common.so.2(+0xa1ebf9) [0x7fb08f21ebf9]
 8: (std::__throw_out_of_range(char const*)+0x41) [0x7fb08eb6ad42]
 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3bc) [0x562c85b03aac]
 10: (MDSMonitor::tick()+0x18a) [0x562c85b1241a]
 11: (Monitor::tick()+0xc0) [0x562c858c9840]
 12: (Context::complete(int)+0xd) [0x562c858e99dd]
 13: (CommonSafeTimer<std::mutex>::timer_thread()+0x12c) [0x7fb08ec1006c]
 14: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7fb08ec11101]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb08f6ac609]
 16: clone()
2023-05-04T08:46:39.612 INFO:scrape:1 jobs: ['7261147']
2023-05-04T08:46:39.613 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2023-05-04T08:46:39.613 INFO:scrape:
2023-05-04T08:46:39.613 INFO:scrape:Failure: Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7fbc054c-e9da-11ed-9b01-001a4aab830c -- ceph mon dump -f json'
2023-05-04T08:46:39.614 INFO:scrape:1 jobs: ['7261084']
2023-05-04T08:46:39.614 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2023-05-04T08:46:39.614 INFO:scrape:
2023-05-04T08:46:39.625 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/
failed:  92
dead:    11
running: 0
waiting: 0
queued:  0
passed:  100


Fail
=================================================================
[7261006]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   02:00:21
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261006/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261006/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1c1add739e24c03927ad0adc400f9f6

    saw valgrind issues

[7261007]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:17:04
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261007/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261007/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd32f61b0e0f47a4a4530e70eff2b4eb

    Test failure: test_volume_ls (tasks.cephfs.test_volumes.TestVolumes)

[7261012]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:18:50
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261012/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261012/
sentry: https://sentry.ceph.com/organizations/ceph/?query=654d3329e6c345429b569a98581347e5

    Command failed (workunit test fs/quota/quota.sh) on smithi187 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'

[7261014]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:26:34
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261014/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261014/

    Command failed on smithi096 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7261019]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
-----------------------------------------------------------------
time:   00:21:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261019/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261019/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c75b6f8faa4a444ebe7d679b680f746c

    Command failed (workunit test fs/fscrypt.sh) on smithi174 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb'

[7261020]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:27:29
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261020/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261020/
sentry: https://sentry.ceph.com/organizations/ceph/?query=09e360e5af3044ab8deafa1e326e5c9a

    Test failure: test_add_data_pool_root
    (tasks.cephfs.test_admin.TestAddDataPool)

[7261023]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
-----------------------------------------------------------------
time:   00:13:50
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261023/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261023/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a5050ef6a154901857e427799b1142b

    Command failed on smithi077 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261026]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:21:51
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261026/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261026/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f5cea8d9d3df4f1387160f30e9cfcbdc

    Test failure: test_random (tasks.cephfs.test_dump_tree.TestDumpTree),
    test_random (tasks.cephfs.test_dump_tree.TestDumpTree)

[7261027]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:09:34
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261027/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261027/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8d410986ea7446238e2b15013113240f

    Command failed on smithi012 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261028]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:22:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261028/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261028/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bba8debefd11454999ea3b66fc99790f

    Command failed on smithi156 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp'

[7261029]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
-----------------------------------------------------------------
time:   00:13:48
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261029/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261029/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35b725a9938a4fe38b07f458a44b54f2

    Command failed on smithi174 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261035]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:09:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261035/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261035/
sentry: https://sentry.ceph.com/organizations/ceph/?query=675f48a4247c426a81e7a06080740dde

    Command failed on smithi129 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261036]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:49:12
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261036/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261036/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd4a5bc30c5e4267a981d79d99d3e8af

    Test failure: test_file_filesystem_sync_crash
    (tasks.cephfs.test_misc.TestMisc), test_file_filesystem_sync_crash
    (tasks.cephfs.test_misc.TestMisc)

[7261037]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:19:01
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261037/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261037/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6697a6e5b2b74f418f3c9049ea29e9c5

    Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent
    (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)

[7261038]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:09:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261038/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261038/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e90ef22cce364e25bca1d9ab3099e4a0

    Command failed on smithi047 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261039]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:39:46
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261039/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261039/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97f567e2168441daa5bddb04d53d5201

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi153 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7261040]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:09:29
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261040/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261040/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c641711c876247069f9d938715c914f6

    Command failed on smithi112 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261043]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
-----------------------------------------------------------------
time:   00:13:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261043/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261043/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cdec2758d4de47ca83e94e8671461acf

    Command failed on smithi096 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261045]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:31:59
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261045/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261045/
sentry: https://sentry.ceph.com/organizations/ceph/?query=057d23b05a58454eb61609c1c69fdcbf

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi118
    with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7261046]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:09:21
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261046/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261046/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01d575f28d7047d8b43c977a5746125a

    Command failed on smithi099 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261049]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:26:02
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261049/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261049/

    Found coredumps on ubuntu@smithi077.front.sepia.ceph.com

[7261050]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:24:30
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261050/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261050/

    Command failed on smithi093 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph quorum_status'

[7261054]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:09:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261054/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261054/
sentry: https://sentry.ceph.com/organizations/ceph/?query=393ade6e82be4a4e95110dcd652211b9

    Command failed on smithi038 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261055]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
-----------------------------------------------------------------
time:   00:11:22
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261055/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261055/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c509893abab4a238402b99bbb646705

    Command failed on smithi090 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7261058]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
-----------------------------------------------------------------
time:   00:13:46
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261058/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261058/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6eef7ddfd3f54f5884b273d610ed9bde

    Command failed on smithi099 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261059]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:32:53
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261059/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261059/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4e01cc2e8fe4515b4ddb42a3a9a9729

    Command failed on smithi012 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'

[7261060]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
-----------------------------------------------------------------
time:   00:17:24
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261060/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261060/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c59f6fb8b4884af49333c5dce8f59d0a

    Test failure: test_dentry_first_precommit
    (tasks.cephfs.test_damage.TestDamage)

[7261065]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   00:37:15
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261065/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261065/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6127b27249b34d54998636fc23bb6597

    Test failure: test_rebuild_inotable
    (tasks.cephfs.test_data_scan.TestDataScan)

[7261068]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
-----------------------------------------------------------------
time:   00:09:23
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261068/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261068/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58babbf52a9d42fca02b55c4d09eda19

    Command failed on smithi107 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261070]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:15:20
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261070/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=233e72e00cf642178a41c2a18aabaee4

    Command failed on smithi162 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7261071]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:23:22
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261071/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f3051902e6154b3ea95f8dc20edcd4fb

    Test failure: test_join_fs_vanilla
    (tasks.cephfs.test_failover.TestClusterAffinity), test_join_fs_vanilla
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7261073]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:17:52
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261073/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261073/

    Found coredumps on ubuntu@smithi093.front.sepia.ceph.com

[7261075]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:27:16
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261075/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261075/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a448dc191271404c9182463a4f006281

    Command failed (workunit test suites/ffsb.sh) on smithi052 with status 135:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7261076]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:52:20
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261076/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261076/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7261077]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:09:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261077/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261077/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5dca32049c9243219e56919b2040a396

    Command failed on smithi053 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261079]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:18:52
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261079/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261079/
sentry: https://sentry.ceph.com/organizations/ceph/?query=62a625ec8a5e47e198062a3f977222ca

    Test failure: test_rapid_creation
    (tasks.cephfs.test_fragment.TestFragmentation)

[7261081]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:59:51
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261081/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261081/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c3ee0f31d3449d8a53d25ba5b85df6e

    saw valgrind issues

[7261083]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:29:23
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261083/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261083/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3ce209e0f9e4ce6b08fa7ce1b9c8c1c

    Test failure: test_subvolume_ops_on_nonexistent_vol
    (tasks.cephfs.test_volumes.TestMisc)

[7261084]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:25
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261084/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261084/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a44011b544b49adb41ad50411c49cee

    Command failed on smithi133 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7fbc054c-e9da-11ed-9b01-001a4aab830c -- ceph mon dump -f json'

[7261085]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:32:18
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261085/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261085/

    Command failed on smithi186 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph quorum_status'

[7261086]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:09:30
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261086/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261086/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5fe075972abf44cda91c9511afbbc6fe

    Command failed on smithi196 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261087]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:16:21
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261087/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261087/
sentry: https://sentry.ceph.com/organizations/ceph/?query=954c95a9caa4446696e407b6adbeb261

    Command failed (workunit test fs/test_python.sh) on smithi182 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7261089]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:13:55
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261089/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261089/

    Found coredumps on ubuntu@smithi033.front.sepia.ceph.com

[7261090]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
-----------------------------------------------------------------
time:   00:09:32
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261090/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e72704a074945209eda3aa854cd8006

    Command failed on smithi118 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261094]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:26:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261094/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261094/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d040e058961441583778984c80111d3

    Test failure: test_full_fclose (tasks.cephfs.test_full.TestQuotaFull)

[7261095]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
-----------------------------------------------------------------
time:   00:09:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261095/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261095/
sentry: https://sentry.ceph.com/organizations/ceph/?query=74b7e4a68c4d4d36bba13dbeb9b25a0a

    Command failed on smithi023 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7261100]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:22:07
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261100/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261100/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ec8b4ed0902473abd7badeaedd477d7

    Test failure: test_for_invalid_mds_rank
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[7261102]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:15:38
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261102/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261102/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90c5c41bc4294f6594e35814a5068baf

    Command failed on smithi146 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7261103]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:28:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261103/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261103/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8627dbd5cc0849308a2e0358e27a9ea9

    Test failure: test_rw_with_fsname_and_no_path_in_cap
    (tasks.cephfs.test_multifs_auth.TestMDSCaps)

[7261104]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:09:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261104/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261104/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24ddc68f5a934f0581da1787756d7595

    Command failed on smithi038 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261108]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
-----------------------------------------------------------------
time:   00:17:07
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261108/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261108/
sentry: https://sentry.ceph.com/organizations/ceph/?query=101f32f870b74b90b2b07940e485642c

    Test failure: test_subvolume_group_snapshot_unsupported_status
    (tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots)

[7261109]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:45:52
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261109/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261109/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11bcce156552407d94ec8926daeaa215

    Command failed (workunit test suites/dbench.sh) on smithi017 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7261110]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
-----------------------------------------------------------------
time:   00:09:37
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261110/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261110/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d87c99b7e8624a4c90654a7799136c2c

    Command failed on smithi141 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261112]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:13:49
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261112/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261112/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6df4f0ae5f01406fbd676bba2f771ae9

    Command failed on smithi023 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261113]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:09:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261113/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261113/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e382360271c44592a4879d283c09b67b

    Command failed on smithi061 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261119]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:14:36
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261119/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd89df3eec4b4cdba07a09d1f8e26f7e

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7261121]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:09:15
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261121/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261121/
sentry: https://sentry.ceph.com/organizations/ceph/?query=66f8f592cd744d8db495e41f67accf66

    Command failed on smithi139 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261123]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
-----------------------------------------------------------------
time:   00:09:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261123/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261123/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de1e3a8bc1ae4f86a6866f0f0c442d78

    Command failed on smithi136 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261126]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:20:02
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261126/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261126/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c082e0446e7246b191b0146a02d5f00b

    Command failed (workunit test suites/fsstress.sh) on smithi186 with status
    2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7261128]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
-----------------------------------------------------------------
time:   00:18:30
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261128/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261128/

    Found coredumps on ubuntu@smithi026.front.sepia.ceph.com

[7261129]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:09:19
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261129/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261129/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22d693f381d8404098cfb46e79f0e6e7

    Command failed on smithi167 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261131]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:09:29
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261131/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261131/
sentry: https://sentry.ceph.com/organizations/ceph/?query=47dff1d7ee9e4f1082e89867894f88b8

    Command failed on smithi182 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261132]  fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
-----------------------------------------------------------------
time:   00:09:49
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261132/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261132/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b865b402eb464a4ab28a82a27163f486

    Command failed on smithi077 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7261134]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
-----------------------------------------------------------------
time:   00:22:48
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261134/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261134/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6388159b95d64a1e9a716c9d6659165c

    Test failure: test_fscrypt_dummy_encryption_with_quick_group
    (tasks.cephfs.test_fscrypt.TestFscrypt),
    test_fscrypt_dummy_encryption_with_quick_group
    (tasks.cephfs.test_fscrypt.TestFscrypt)

[7261135]  fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
-----------------------------------------------------------------
time:   00:09:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261135/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261135/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c9a8d918157f455583656158b1f9e09c

    Command failed on smithi175 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7261137]  fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
-----------------------------------------------------------------
time:   00:13:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261137/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261137/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40085040fd334a0cb33c6bb9f0160cba

    Command failed on smithi093 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7261139]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:25:02
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261139/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261139/
sentry: https://sentry.ceph.com/organizations/ceph/?query=21ffadbe0427427fa1585516088e789a

    Test failure: test_dump_inmemory_log_on_missed_beacon_ack_from_monitors
    (tasks.cephfs.test_misc.TestMisc)

[7261140]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:09:35
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261140/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261140/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19f09973818e48368dcb29d458f1887c

    Command failed on smithi039 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261143]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:13:54
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261143/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261143/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b1471cb19e24711988ee229a01eaa16

    Command failed on smithi194 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261145]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:09:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261145/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e51be8858fc430c9d9af7bbaf78ca6e

    Command failed on smithi081 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-
    immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo
    ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-
    debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo
    libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-
    debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-
    debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo
    python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel
    libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-
    devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server
    postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib'

[7261147]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:31:18
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261147/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e71ecd4d7f3649768572661cbbd932ab

    Test failure: test_concurrent_snap_creates
    (tasks.cephfs.test_snap_schedules.TestSnapSchedules),
    test_concurrent_snap_creates
    (tasks.cephfs.test_snap_schedules.TestSnapSchedules)

[7261151]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
-----------------------------------------------------------------
time:   00:15:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261151/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261151/

    Found coredumps on ubuntu@smithi096.front.sepia.ceph.com

[7261154]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
-----------------------------------------------------------------
time:   00:23:22
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261154/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261154/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be6c36a628dd4418ab0d54027f69ca61

    Test failure: test_mds_max_snaps_per_dir_default_limit
    (tasks.cephfs.test_snapshots.TestSnapshots)

[7261155]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
-----------------------------------------------------------------
time:   00:09:41
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261155/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261155/
sentry: https://sentry.ceph.com/organizations/ceph/?query=881cb333950e46efbba5b44ec4837a8e

    Command failed on smithi007 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261157]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   00:09:23
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261157/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8aededfb17b54ca0a553e5b2a41b9b8b

    Command failed on smithi082 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261159]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:09:23
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261159/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261159/
sentry: https://sentry.ceph.com/organizations/ceph/?query=123c89befd5a4e28b9374b286704f791

    Command failed on smithi084 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261160]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   00:09:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261160/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261160/
sentry: https://sentry.ceph.com/organizations/ceph/?query=87ecd592bbdd49269c4e4da6de4dbe0d

    Command failed on smithi101 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261161]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:24:50
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261161/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261161/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee664477d3c74465b61fe7d320db480d

    Test failure: test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration),
    test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration)

[7261164]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:59:42
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261164/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261164/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d85b05aeb05046e1b70bdf443a7016f1

    saw valgrind issues

[7261170]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:31:39
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261170/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261170/
sentry: https://sentry.ceph.com/organizations/ceph/?query=83e81458b7b04afb973a7ac995d2dd52

    Test failure: test_clone_subvolume_info
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7261174]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:15:36
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261174/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261174/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22a7864392cd4722b8cf9291a64834cc

    Command failed on smithi135 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7261175]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:19:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261175/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261175/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b94e66b577742e2b488fa14c55137fd

    Test failure: test_join_fs_steady
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7261178]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
-----------------------------------------------------------------
time:   00:09:43
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261178/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261178/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6fd3b1fa6cb84d31a541acdceac6318f

    Command failed on smithi047 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261182]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:16:19
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261182/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261182/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5edcb854210e4ab9b0d5f67611319f49

    Command failed (workunit test fs/quota/quota.sh) on smithi142 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd --
    /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3
    CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'

[7261188]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:26:33
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261188/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261188/
sentry: https://sentry.ceph.com/organizations/ceph/?query=757d8dfd597e44f6ba9567d355a2d4bb

    Test failure: test_standby_compat
    (tasks.cephfs.test_admin.TestCompatCommands)

[7261191]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
-----------------------------------------------------------------
time:   00:09:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261191/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261191/
sentry: https://sentry.ceph.com/organizations/ceph/?query=03b915c2f52c404da22558121fc72ca8

    Command failed on smithi181 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261197]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:23:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261197/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261197/
sentry: https://sentry.ceph.com/organizations/ceph/?query=269fdde810724a048f09569c6966c1ad

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi037
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=221532d2ce1c467630b0dec52a6c3ecd8d441799
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7261199]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:13:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261199/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261199/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7562c3d4494d4d1499a38677b882400b

    Command failed on smithi087 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib'

[7261203]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
-----------------------------------------------------------------
time:   00:18:43
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261203/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261203/
sentry: https://sentry.ceph.com/organizations/ceph/?query=473b083a73ad4ce89c6d110eeec3c3dc

    Test failure: test_mds_readonly
    (tasks.cephfs.test_auto_repair.TestMDSAutoRepair)

[7261206]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
-----------------------------------------------------------------
time:   00:09:32
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261206/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261206/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06376f6311254a9fbfde8e17cef98e12

    Command failed on smithi042 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr-
    diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2
    libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados
    python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel
    openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux'

[7261207]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:14:29
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261207/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261207/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4f9d428b1f9d4463b60f742e551b12ca

    Command failed on smithi141 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7261208]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:23:11
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261208/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261208/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80f6cb8df529452cbc9b81ad1b98102c

    Test failure: test_r_with_no_fsname_and_path_in_cap
    (tasks.cephfs.test_multifs_auth.TestMDSCaps)



Dead
=================================================================
[7261009]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261009/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261009/

    hit max job timeout

[7261024]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261024/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261024/

    hit max job timeout

[7261080]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261080/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261080/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7261098]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261098/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261098/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7261111]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261111/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261111/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7261120]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261120/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261120/

    hit max job timeout

[7261125]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261125/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261125/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7261138]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261138/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261138/

    hit max job timeout

[7261142]  fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261142/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261142/

    hit max job timeout

[7261146]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261146/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261146/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds

[7261177]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261177/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261177/

    hit max job timeout



Pass
=================================================================
[7261008] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:48:17
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261008/

[7261010] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:32:58
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261010/

[7261011] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:02:20
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261011/

[7261013] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:41:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261013/

[7261015] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:29:11
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261015/

[7261016] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:18:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261016/

[7261017] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:46:21
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261017/

[7261018] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:27:56
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261018/

[7261021] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:25:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261021/

[7261022] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:34:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261022/

[7261025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   01:01:42
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261025/

[7261030] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:14:15
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261030/

[7261031] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
time:   00:26:03
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261031/

[7261032] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:14:22
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261032/

[7261033] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:24:02
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261033/

[7261034] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:26:37
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261034/

[7261041] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:35:53
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261041/

[7261042] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:37:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261042/

[7261044] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:19:02
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261044/

[7261047] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:16:45
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261047/

[7261048] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
time:   00:39:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261048/

[7261051] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:43:21
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261051/

[7261052] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:13:57
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261052/

[7261053] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:25:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261053/

[7261056] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261056/

[7261057] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
time:   00:26:35
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261057/

[7261061] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:10
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261061/

[7261062] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:18:06
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261062/

[7261063] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:24:24
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261063/

[7261064] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:37:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261064/

[7261066] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:44:38
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261066/

[7261067] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
time:   01:20:38
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261067/

[7261069] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:05
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261069/

[7261072] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:52
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261072/

[7261074] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:37:06
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261074/

[7261078] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:50:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261078/

[7261082] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:25:17
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261082/

[7261088] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:34
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261088/

[7261091] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}}
time:   01:25:32
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261091/

[7261092] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:58:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261092/

[7261093] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:54
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261093/

[7261096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:14:13
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261096/

[7261097] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:18:36
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261097/

[7261099] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261099/

[7261101] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:19:00
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261101/

[7261105] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:21:03
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261105/

[7261106] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:23:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261106/

[7261107] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:31:04
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261107/

[7261114] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:14:01
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261114/

[7261115] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:30:42
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261115/

[7261116] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:58:48
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261116/

[7261117] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:44:05
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261117/

[7261118] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:02:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261118/

[7261122] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:53:58
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261122/

[7261124] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261124/

[7261127] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}}
time:   00:47:27
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261127/

[7261130] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:47:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261130/

[7261133] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:57:25
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261133/

[7261136] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:17:44
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261136/

[7261141] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:20
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261141/

[7261144] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:23:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261144/

[7261148] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:06
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261148/

[7261149] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:24:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261149/

[7261150] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
time:   00:36:48
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261150/

[7261152] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:14:46
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261152/

[7261153] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:32:52
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261153/

[7261156] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:20:55
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261156/

[7261158] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}}
time:   00:42:06
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261158/

[7261162] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:17:28
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261162/

[7261163] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:45:47
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261163/

[7261165] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:25:17
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261165/

[7261166] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:26
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261166/

[7261167] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:15:25
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261167/

[7261168] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:30:40
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261168/

[7261169] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:28:01
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261169/

[7261171] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:32:08
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261171/

[7261172] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:17:12
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261172/

[7261173] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:48:38
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261173/

[7261176] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:51:30
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261176/

[7261179] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:41:14
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261179/

[7261180] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:23:42
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261180/

[7261181] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:04
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261181/

[7261183] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
time:   01:11:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261183/

[7261184] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:21:50
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261184/

[7261185] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:41:50
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261185/

[7261186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:17:19
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261186/

[7261187] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
time:   00:38:15
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261187/

[7261189] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:07:09
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261189/

[7261190] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:08
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261190/

[7261192] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:51
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261192/

[7261193] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:14:11
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261193/

[7261194] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:23
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261194/

[7261195] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:31
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261195/

[7261196] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:24:09
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261196/

[7261198] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:49
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261198/

[7261200] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}}
time:   00:43:12
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261200/

[7261201] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:34:29
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261201/

[7261202] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:38
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261202/

[7261204] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep}
time:   00:20:37
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261204/

[7261205] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:22:30
info:   http://pulpito.ceph.com/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261205/