2024-03-14T15:22:58.748 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T15:22:58.749 INFO:teuthology.results:subset: '383/512'
2024-03-14T15:22:58.749 INFO:teuthology.results:seed: '6353'
2024-03-14T16:54:45.206 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T16:54:45.242 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-14T22:55:13.204 INFO:teuthology.results:Tests finished! gathering results...
2024-03-14T22:55:19.158 INFO:scrape:Found 70 jobs
2024-03-14T22:55:48.672 INFO:scrape:Found 14 distinct failure reasons
2024-03-14T22:55:48.673 INFO:scrape:Failure: Command failed on smithi003 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub'
2024-03-14T22:55:48.673 INFO:scrape:3 jobs: ['7600240', '7600210', '7600189']
2024-03-14T22:55:48.673 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus']
2024-03-14T22:55:48.673 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus']
2024-03-14T22:55:48.673 INFO:scrape:
2024-03-14T22:55:48.673 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-03-14T22:55:48.673 INFO:scrape:5 jobs: ['7600229', '7600186', '7600243', '7600201', '7600221']
2024-03-14T22:55:48.674 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-03-14T22:55:48.674 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}']
2024-03-14T22:55:48.674 INFO:scrape:
2024-03-14T22:55:48.674 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2024-03-14T22:55:48.674 INFO:scrape:2 jobs: ['7600191', '7600250']
2024-03-14T22:55:48.674 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2024-03-14T22:55:48.674 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2024-03-14T22:55:48.674 INFO:scrape:
2024-03-14T22:55:48.675 INFO:scrape:Failure: valgrind error: Leak_StillReachable
malloc
malloc
strdup
2024-03-14T22:55:48.675 INFO:scrape:2 jobs: ['7600209', '7600231']
2024-03-14T22:55:48.675 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2024-03-14T22:55:48.675 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2024-03-14T22:55:48.675 INFO:scrape:
2024-03-14T22:55:48.675 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi045 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'
2024-03-14T22:55:48.675 INFO:scrape:4 jobs: ['7600212', '7600239', '7600225', '7600197']
2024-03-14T22:55:48.675 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-14T22:55:48.675 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}']
2024-03-14T22:55:48.675 INFO:scrape:
2024-03-14T22:55:48.676 INFO:scrape:Failure: Command failed on smithi044 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf'
2024-03-14T22:55:48.676 INFO:scrape:3 jobs: ['7600215', '7600196', '7600213']
2024-03-14T22:55:48.676 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2024-03-14T22:55:48.676 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/cfuse_workunit_snaptests}}', '3-snaps/no', '4-flush/yes', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'export-check', 'frag', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/workunit/snaps}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2024-03-14T22:55:48.676 INFO:scrape:
2024-03-14T22:55:48.676 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
2024-03-14T22:55:48.676 INFO:scrape:1 jobs: ['7600195']
2024-03-14T22:55:48.676 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}']
2024-03-14T22:55:48.676 INFO:scrape:
2024-03-14T22:55:48.677 INFO:scrape:Failure: Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84fd03b6-e219-11ee-95c9-87774f69a715 -e sha1=d16d1766eb379253122a6acda1cdd17a0b9c3aa3 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''
2024-03-14T22:55:48.677 INFO:scrape:2 jobs: ['7600193', '7600208']
2024-03-14T22:55:48.677 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-14T22:55:48.677 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-14T22:55:48.677 INFO:scrape:
2024-03-14T22:55:48.677 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2024-03-14T22:55:48.677 INFO:scrape:4 jobs: ['7600184', '7600230', '7600202', '7600234']
2024-03-14T22:55:48.677 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/automatic', 'session_timeout}', 'standby-replay']
2024-03-14T22:55:48.677 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-14T22:55:48.677 INFO:scrape:
2024-03-14T22:55:48.678 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts)
2024-03-14T22:55:48.678 INFO:scrape:1 jobs: ['7600194']
2024-03-14T22:55:48.678 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-14T22:55:48.678 INFO:scrape:
2024-03-14T22:55:48.678 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2024-03-14T22:55:48.678 INFO:scrape:1 jobs: ['7600198']
2024-03-14T22:55:48.678 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/xfstests-dev}']
2024-03-14T22:55:48.678 INFO:scrape:
2024-03-14T22:55:48.678 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable)
2024-03-14T22:55:48.678 INFO:scrape:1 jobs: ['7600232']
2024-03-14T22:55:48.679 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/openfiletable}']
2024-03-14T22:55:48.679 INFO:scrape:
2024-03-14T22:55:48.679 INFO:scrape:Failure: Failed to reconnect to smithi112
2024-03-14T22:55:48.679 INFO:scrape:1 jobs: ['7600205']
2024-03-14T22:55:48.679 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2024-03-14T22:55:48.679 INFO:scrape:
2024-03-14T22:55:48.679 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2024-03-14T22:55:48.679 INFO:scrape:1 jobs: ['7600227']
2024-03-14T22:55:48.679 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-14T22:55:48.679 INFO:scrape:
2024-03-14T22:55:48.684 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/
failed:  31
dead:    0
running: 0
waiting: 0
queued:  0
passed:  39


Fail
=================================================================
[7600184]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:54:16
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600184/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600184/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7600186]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:37:24
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600186/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600186/

    reached maximum tries (51) after waiting for 300 seconds

[7600189]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:23:58
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600189/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600189/

    Command failed on smithi007 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 4.2c deep-scrub'

[7600191]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:43:44
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600191/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600191/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0a4f0576ed34fdda5e24d7297f9ca70

    Command failed (workunit test suites/dbench.sh) on smithi059 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7600193]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:58:05
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600193/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600193/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33c32dd142404fa8b45e7ad6524a1892

    Command failed on smithi118 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    84fd03b6-e219-11ee-95c9-87774f69a715 -e
    sha1=d16d1766eb379253122a6acda1cdd17a0b9c3aa3 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7600194]  fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:40:02
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600194/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600194/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6aebb3509bba445a9b0dd55ce2e91dfe

    Test failure: test_reading_conf
    (tasks.cephfs.test_cephfs_shell.TestShellOpts)

[7600195]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:34:44
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600195/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600195/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4af0ca4c38be4bd399c8c4adbdda2973

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
    (tasks.cephfs.test_mirroring.TestMirroring)

[7600196]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:25:12
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600196/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600196/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5290663d14b24391aaebcfdf622189c0

    Command failed on smithi032 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf'

[7600197]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:33:49
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600197/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600197/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c3b2360f6cf44df82ef66af5af091b9

    Command failed (workunit test suites/fsx.sh) on smithi039 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[7600198]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/xfstests-dev}
-----------------------------------------------------------------
time:   07:20:33
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600198/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600198/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e4e858113314551afdcf11347bf1907

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7600201]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:40:03
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600201/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600201/

    reached maximum tries (51) after waiting for 300 seconds

[7600202]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   01:03:30
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600202/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600202/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7600205]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600205/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600205/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a7a543fe8b3e4d038dc7d2fdcbbd171b

    Failed to reconnect to smithi112

[7600208]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:56:43
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600208/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600208/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71988b84fcfa4979804e4fc7073d64f6

    Command failed on smithi049 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a0e80a3a-e219-11ee-95c9-87774f69a715 -e
    sha1=d16d1766eb379253122a6acda1cdd17a0b9c3aa3 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7600209]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:56:12
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600209/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600209/

    valgrind error: Leak_StillReachable malloc malloc strdup

[7600210]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:24:28
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600210/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600210/

    Command failed on smithi088 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 4.3a deep-scrub'

[7600212]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:34:07
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600212/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600212/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2155d545fad749abb4b8a2182a3b945d

    Command failed (workunit test suites/fsx.sh) on smithi045 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[7600213]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:12:03
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600213/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600213/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7bc1465b6a68447c8fa04757cc153434

    Command failed on smithi046 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf'

[7600215]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:12:50
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600215/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600215/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab69065877184eb3b51d18477c6e1025

    Command failed on smithi044 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf'

[7600221]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:54:56
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600221/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600221/

    reached maximum tries (51) after waiting for 300 seconds

[7600225]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:26:00
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600225/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600225/
sentry: https://sentry.ceph.com/organizations/ceph/?query=920980d8080d460d8d67d35ee3d337f6

    Command failed (workunit test suites/fsx.sh) on smithi002 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[7600227]  fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:12:51
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600227/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600227/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4832ade948c44c359f247768ca3c4d60

    Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)

[7600229]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:07:28
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600229/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600229/

    reached maximum tries (51) after waiting for 300 seconds

[7600230]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   01:53:32
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600230/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600230/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7600231]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:44:35
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600231/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600231/

    valgrind error: Leak_StillReachable malloc malloc strdup

[7600232]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/openfiletable}
-----------------------------------------------------------------
time:   00:15:43
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600232/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600232/
sentry: https://sentry.ceph.com/organizations/ceph/?query=21abe981cee94de8b94274ebf5efe608

    Test failure: test_max_items_per_obj
    (tasks.cephfs.test_openfiletable.OpenFileTable)

[7600234]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   01:09:25
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600234/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600234/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7600239]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:26:36
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600239/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600239/
sentry: https://sentry.ceph.com/organizations/ceph/?query=862c2cef26cf4c2c834dafdb284157ae

    Command failed (workunit test suites/fsx.sh) on smithi086 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[7600240]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:23:35
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600240/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600240/

    Command failed on smithi003 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 4.3a deep-scrub'

[7600243]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:23
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600243/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600243/

    reached maximum tries (51) after waiting for 300 seconds

[7600250]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:38:45
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600250/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600250/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67ca0ceab2dc4e2dbcc8c094c5e69b39

    Command failed (workunit test suites/dbench.sh) on smithi046 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f9614ef45ac4fdf44b6e9d8fb76c8de50ffecfaf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'



Pass
=================================================================
[7600182] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:36:33
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600182/

[7600183] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:16:45
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600183/

[7600185] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snapshots}
time:   00:23:31
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600185/

[7600187] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}}
time:   01:03:33
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600187/

[7600188] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays}
time:   00:31:35
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600188/

[7600190] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:36:24
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600190/

[7600192] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:36:59
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600192/

[7600199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/acls}
time:   00:16:58
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600199/

[7600200] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
time:   00:30:01
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600200/

[7600203] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool}
time:   00:12:28
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600203/

[7600204] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:20:30
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600204/

[7600206] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:07:27
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600206/

[7600207] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:54:17
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600207/

[7600211] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:13:31
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600211/

[7600214] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-recovery}
time:   00:54:37
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600214/

[7600216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage}
time:   00:25:30
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600216/

[7600217] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
time:   00:54:30
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600217/

[7600218] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan}
time:   00:37:44
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600218/

[7600219] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
time:   02:00:44
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600219/

[7600220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/exports}
time:   00:50:10
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600220/

[7600222] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub}
time:   00:24:09
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600222/

[7600223] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:21:10
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600223/

[7600224] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:43:11
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600224/

[7600226] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:09:40
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600226/

[7600228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/multimds_misc}
time:   00:14:34
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600228/

[7600233] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:11
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600233/

[7600235] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:12:17
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600235/

[7600236] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:43:48
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600236/

[7600237] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:20:07
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600237/

[7600238] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:31:49
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600238/

[7600241] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub}
time:   00:21:05
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600241/

[7600242] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/sessionmap}
time:   00:17:21
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600242/

[7600244] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
time:   00:50:53
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600244/

[7600245] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:53:10
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600245/

[7600246] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}}
time:   01:05:57
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600246/

[7600247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snapshots}
time:   00:26:39
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600247/

[7600248] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:13:49
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600248/

[7600249] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:58:20
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600249/

[7600251] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}
time:   00:27:38
info:   https://pulpito.ceph.com/pdonnell-2024-03-14_15:20:35-fs-wip-batrick-testing-20240310.014948-distro-default-smithi/7600251/