2024-03-26T05:49:09.330 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-26T05:49:09.330 INFO:teuthology.results:subset: '481/512'
2024-03-26T05:49:09.330 INFO:teuthology.results:seed: '1249'
2024-03-26T06:58:24.674 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-26T06:58:24.697 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-26T13:33:54.492 INFO:teuthology.results:Tests finished! gathering results...
2024-03-26T13:33:59.287 INFO:scrape:Found 55 jobs
2024-03-26T13:34:01.763 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.763 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.763 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.763 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.850 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.851 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.851 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.851 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.917 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.917 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.918 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.918 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.983 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.983 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.983 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:01.983 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.050 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.051 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.051 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.051 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.435 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.435 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.435 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.435 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.645 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.645 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.646 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.646 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.821 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.821 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.821 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.822 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.987 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.988 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.988 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:02.988 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.149 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.149 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.149 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.149 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.322 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.322 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.322 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:03.322 WARNING:scrape:Saw end of BT but not start
2024-03-26T13:34:21.883 INFO:scrape:Found 14 distinct failure reasons
2024-03-26T13:34:21.884 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-03-26T13:34:21.884 INFO:scrape:5 jobs: ['7622764', '7622755', '7622745', '7622773', '7622741']
2024-03-26T13:34:21.884 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-26T13:34:21.885 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-26T13:34:21.885 INFO:scrape:
2024-03-26T13:34:21.885 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi033 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'
2024-03-26T13:34:21.885 INFO:scrape:5 jobs: ['7622791', '7622776', '7622765', '7622754', '7622742']
2024-03-26T13:34:21.885 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-26T13:34:21.885 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-26T13:34:21.885 INFO:scrape:
2024-03-26T13:34:21.885 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2024-03-26T13:34:21.886 INFO:scrape:1 jobs: ['7622752']
2024-03-26T13:34:21.886 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2024-03-26T13:34:21.886 INFO:scrape:
2024-03-26T13:34:21.886 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2024-03-26T13:34:21.886 INFO:scrape:2 jobs: ['7622788', '7622762']
2024-03-26T13:34:21.886 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/automatic', 'session_timeout}', 'standby-replay']
2024-03-26T13:34:21.886 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2024-03-26T13:34:21.886 INFO:scrape:
2024-03-26T13:34:21.887 INFO:scrape:Failure: Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4fe4dba6-eb40-11ee-95d0-87774f69a715 -e sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2024-03-26T13:34:21.887 INFO:scrape:1 jobs: ['7622792']
2024-03-26T13:34:21.887 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}']
2024-03-26T13:34:21.887 INFO:scrape:
2024-03-26T13:34:21.887 INFO:scrape:Failure: valgrind error: Leak_StillReachable
operator new[](unsigned long)
rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr<rocksdb::ObjectLibrary::Entry, std::default_delete<rocksdb::ObjectLibrary::Entry> >&&)
UnknownInlinedFun
2024-03-26T13:34:21.887 INFO:scrape:1 jobs: ['7622751']
2024-03-26T13:34:21.887 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}']
2024-03-26T13:34:21.887 INFO:scrape:
2024-03-26T13:34:21.888 INFO:scrape:Crash: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev), test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2024-03-26T13:34:21.888 INFO:scrape:ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7fe5e1854db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7fe5e18a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x187) [0x5556a08db56c]
 6: (KernelDevice::_aio_thread()+0x11e0) [0x5556a12ac3f0]
 7: ceph-osd(+0xdc6561) [0x5556a12ac561]
 8: /lib64/libc.so.6(+0x9f802) [0x7fe5e189f802]
 9: /lib64/libc.so.6(+0x3f450) [0x7fe5e183f450]
2024-03-26T13:34:21.888 INFO:scrape:1 jobs: ['7622744']
2024-03-26T13:34:21.888 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}']
2024-03-26T13:34:21.888 INFO:scrape:
2024-03-26T13:34:21.888 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2024-03-26T13:34:21.888 INFO:scrape:1 jobs: ['7622756']
2024-03-26T13:34:21.888 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-26T13:34:21.888 INFO:scrape:
2024-03-26T13:34:21.888 INFO:scrape:Failure: Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2024-03-26T13:34:21.889 INFO:scrape:2 jobs: ['7622779', '7622784']
2024-03-26T13:34:21.889 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
2024-03-26T13:34:21.889 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}']
2024-03-26T13:34:21.889 INFO:scrape:
2024-03-26T13:34:21.889 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts)
2024-03-26T13:34:21.889 INFO:scrape:1 jobs: ['7622782']
2024-03-26T13:34:21.889 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-26T13:34:21.889 INFO:scrape:
2024-03-26T13:34:21.889 INFO:scrape:Failure: Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi031 with status 11: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'
2024-03-26T13:34:21.889 INFO:scrape:1 jobs: ['7622785']
2024-03-26T13:34:21.889 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2024-03-26T13:34:21.890 INFO:scrape:
2024-03-26T13:34:21.890 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2024-03-26T13:34:21.890 INFO:scrape:3 jobs: ['7622789', '7622740', '7622772']
2024-03-26T13:34:21.890 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-26T13:34:21.890 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-26T13:34:21.890 INFO:scrape:
2024-03-26T13:34:21.890 INFO:scrape:Failure: Test failure: test_fs_rename_fails_for_non_existent_fs (tasks.cephfs.test_admin.TestRenameCommand)
2024-03-26T13:34:21.890 INFO:scrape:1 jobs: ['7622749']
2024-03-26T13:34:21.890 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/admin}']
2024-03-26T13:34:21.891 INFO:scrape:
2024-03-26T13:34:21.891 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost
calloc
__trans_list_add

2024-03-26T13:34:21.891 INFO:scrape:1 jobs: ['7622768']
2024-03-26T13:34:21.891 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}']
2024-03-26T13:34:21.891 INFO:scrape: