2024-04-01T14:15:32.177 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-01T14:15:32.177 INFO:teuthology.results:subset: '932/1024'
2024-04-01T14:15:32.177 INFO:teuthology.results:seed: '6891'
2024-04-01T16:15:25.780 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-01T16:15:25.804 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-02T03:58:22.949 INFO:teuthology.results:Tests finished! gathering results...
2024-04-02T03:58:30.213 INFO:scrape:Found 75 jobs
2024-04-02T03:58:59.243 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633901/teuthology.log
2024-04-02T03:58:59.243 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633901/teuthology.log
2024-04-02T03:59:05.832 INFO:scrape:Found 16 distinct failure reasons
2024-04-02T03:59:05.832 INFO:scrape:Failure: Command failed on smithi029 with status 5: 'sudo systemctl stop ceph-838a9dac-f044-11ee-b647-cb9ed24678a4@mon.a'
2024-04-02T03:59:05.832 INFO:scrape:36 jobs
2024-04-02T03:59:05.833 INFO:scrape:suites intersection: []
2024-04-02T03:59:05.833 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '1-rename', '2-logrotate', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '3-modules}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'distro/single-container-host}', 'export-check', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-04-02T03:59:05.833 INFO:scrape:
2024-04-02T03:59:05.833 INFO:scrape:Failure: Command failed on smithi038 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3de72798-f044-11ee-b647-cb9ed24678a4 -e sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"
2024-04-02T03:59:05.834 INFO:scrape:8 jobs: ['7633960', '7633937', '7633925', '7633945', '7633896', '7633951', '7633902', '7633931']
2024-04-02T03:59:05.834 INFO:scrape:suites intersection: ['1-volume/{0-create', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-04-02T03:59:05.834 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}']
2024-04-02T03:59:05.834 INFO:scrape:
2024-04-02T03:59:05.834 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2024-04-02T03:59:05.834 INFO:scrape:4 jobs: ['7633954', '7633943', '7633926', '7633897']
2024-04-02T03:59:05.834 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cfuse_workunit_suites_pjd}']
2024-04-02T03:59:05.835 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cfuse_workunit_suites_pjd}']
2024-04-02T03:59:05.835 INFO:scrape:
2024-04-02T03:59:05.835 INFO:scrape:Failure: SSH connection to smithi163 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=-'
2024-04-02T03:59:05.835 INFO:scrape:2 jobs: ['7633911', '7633912']
2024-04-02T03:59:05.835 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}']
2024-04-02T03:59:05.835 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-logrotate', '2-scrub/no', '3-inline/no', '3-modules}', '3-snaps/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/test_o_trunc}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'fail_fs/no', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2024-04-02T03:59:05.835 INFO:scrape:
2024-04-02T03:59:05.835 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts)
2024-04-02T03:59:05.835 INFO:scrape:1 jobs: ['7633916']
2024-04-02T03:59:05.836 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}']
2024-04-02T03:59:05.836 INFO:scrape:
2024-04-02T03:59:05.836 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
2024-04-02T03:59:05.836 INFO:scrape:1 jobs: ['7633917']
2024-04-02T03:59:05.836 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}']
2024-04-02T03:59:05.836 INFO:scrape:
2024-04-02T03:59:05.836 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2024-04-02T03:59:05.836 INFO:scrape:1 jobs: ['7633920']
2024-04-02T03:59:05.836 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}']
2024-04-02T03:59:05.836 INFO:scrape:
2024-04-02T03:59:05.836 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2024-04-02T03:59:05.837 INFO:scrape:1 jobs: ['7633941']
2024-04-02T03:59:05.837 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/xfstests-dev}']
2024-04-02T03:59:05.837 INFO:scrape:
2024-04-02T03:59:05.837 INFO:scrape:Failure: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3faca88-f03a-11ee-b647-cb9ed24678a4 -e sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\''
2024-04-02T03:59:05.837 INFO:scrape:1 jobs: ['7633907']
2024-04-02T03:59:05.837 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-04-02T03:59:05.837 INFO:scrape:
2024-04-02T03:59:05.837 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable)
2024-04-02T03:59:05.837 INFO:scrape:1 jobs: ['7633918']
2024-04-02T03:59:05.837 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/openfiletable}']
2024-04-02T03:59:05.838 INFO:scrape:
2024-04-02T03:59:05.838 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0367380f94]
2024-04-02T03:59:05.838 INFO:scrape:ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f0366a54db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f0366aa154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0367380f94]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x1810f8) [0x7f03673810f8]
 7: (Inode::put_cap_ref(int)+0x313) [0x564e0cb5c643]
 8: (Client::put_cap_ref(Inode*, int)+0x32) [0x564e0cad74e2]
 9: (Client::C_Read_Async_Finisher::finish(int)+0x21) [0x564e0cb139a1]
 10: ceph_test_client(+0xb7fad) [0x564e0ca8bfad]
 11: (Client::_read_async(Fh*, unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, Context*)+0xdc) [0x564e0cb13abc]
 12: (Client::_read(Fh*, long, unsigned long, ceph::buffer::v15_2_0::list*, Context*)+0x25c) [0x564e0cb14a7c]
 13: (Client::_preadv_pwritev_locked(Fh*, iovec const*, int, long, bool, bool, Context*, ceph::buffer::v15_2_0::list*, bool, bool)+0x10d) [0x564e0cb1acdd]
 14: (Client::ll_preadv_pwritev(Fh*, iovec const*, int, long, bool, Context*, ceph::buffer::v15_2_0::list*, bool, bool)+0x2d2) [0x564e0cb3d822]
 15: (TestClient_LlreadvLlwritevReadOnlyFile_Test::TestBody()+0x431) [0x564e0caa5091]
 16: ceph_test_client(+0x2461ec) [0x564e0cc1a1ec]
 17: (testing::Test::Run()+0xe3) [0x564e0cb85d43]
 18: ceph_test_client(+0x1d1165) [0x564e0cba5165]
 19: ceph_test_client(+0x1d12f9) [0x564e0cba52f9]
 20: (testing::internal::UnitTestImpl::RunAllTests()+0x804) [0x564e0cb97604]
 21: (testing::UnitTest::Run()+0x1a8) [0x564e0cb939a8]
 22: main()
 23: /lib64/libc.so.6(+0x3feb0) [0x7f0366a3feb0]
 24: __libc_start_main()
 25: _start()
2024-04-02T03:59:05.838 INFO:scrape:1 jobs: ['7633914']
2024-04-02T03:59:05.838 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/client}']
2024-04-02T03:59:05.838 INFO:scrape:
2024-04-02T03:59:05.838 INFO:scrape:Dead: b'2024-04-02T03:54:51.964 DEBUG:teuthology.exit:Finished running handlers'
2024-04-02T03:59:05.838 INFO:scrape:1 jobs: ['7633947']
2024-04-02T03:59:05.839 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/admin}']
2024-04-02T03:59:05.839 INFO:scrape:
2024-04-02T03:59:05.839 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost
calloc
__trans_list_add

2024-04-02T03:59:05.839 INFO:scrape:2 jobs: ['7633909', '7633958']
2024-04-02T03:59:05.839 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'session_timeout}', 'validater/valgrind}']
2024-04-02T03:59:05.839 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'tasks/fsstress', 'validater/valgrind}']
2024-04-02T03:59:05.839 INFO:scrape:
2024-04-02T03:59:05.839 INFO:scrape:Dead: None
2024-04-02T03:59:05.839 INFO:scrape:1 jobs: ['7633901']
2024-04-02T03:59:05.839 INFO:scrape:
2024-04-02T03:59:05.840 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2024-04-02T03:59:05.840 INFO:scrape:1 jobs: ['7633955']
2024-04-02T03:59:05.840 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}']
2024-04-02T03:59:05.840 INFO:scrape:
2024-04-02T03:59:05.840 INFO:scrape:Failure: "2024-04-01T16:20:25.282256+0000 mon.a (mon.0) 410 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2024-04-02T03:59:05.840 INFO:scrape:1 jobs: ['7633942']
2024-04-02T03:59:05.840 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}']
2024-04-02T03:59:05.840 INFO:scrape:
2024-04-02T03:59:05.845 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/
failed:  59
dead:    4
running: 0
waiting: 0
queued:  0
passed:  12


Fail
=================================================================
[7633892]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:08:26
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633892/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=943dbf5c4804495387ab55cb80443ef9

    Command failed on smithi080 with status 5: 'sudo systemctl stop
    ceph-c7be1eb4-f034-11ee-b647-cb9ed24678a4@mon.a'

[7633893]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:10:23
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633893/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633893/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24d0d1f5b984461790c4ba7d6bac47e1

    Command failed on smithi100 with status 5: 'sudo systemctl stop
    ceph-56884dcc-f035-11ee-b647-cb9ed24678a4@mon.a'

[7633895]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:08:36
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633895/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633895/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d99253d8b6f400685d4b670bcc31afe

    Command failed on smithi052 with status 5: 'sudo systemctl stop
    ceph-9aec4d50-f036-11ee-b647-cb9ed24678a4@mon.a'

[7633896]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:02
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633896/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b091c2533843412094adf254853b3ccc

    Command failed on smithi047 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    faae04e0-f036-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633897]  fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:15:52
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633897/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633897/
sentry: https://sentry.ceph.com/organizations/ceph/?query=346104202c8d462095bb25d958119650

    Command failed (workunit test suites/pjd.sh) on smithi167 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7633898]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:09:39
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633898/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633898/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e50b74412a841d79ccbfb43431eb1ae

    Command failed on smithi071 with status 5: 'sudo systemctl stop
    ceph-3f7aae2a-f037-11ee-b647-cb9ed24678a4@mon.a'

[7633899]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:07:59
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633899/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633899/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8427b406ef50462a87cc46b2ed8bd1a1

    Command failed on smithi138 with status 5: 'sudo systemctl stop
    ceph-6c306cde-f037-11ee-b647-cb9ed24678a4@mon.a'

[7633900]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:10:47
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633900/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633900/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d5e1513ef3e143da9d384474f5f7e391

    Command failed on smithi019 with status 5: 'sudo systemctl stop
    ceph-42bedf82-f039-11ee-b647-cb9ed24678a4@mon.a'

[7633902]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:27:03
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633902/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4847a6d133c4ca8891e7bdbac435d06

    Command failed on smithi029 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    52dea294-f039-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633904]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:09:55
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633904/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633904/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1f60208d9a4247d7940eb0667101f1c9

    Command failed on smithi088 with status 5: 'sudo systemctl stop
    ceph-1c1ea222-f039-11ee-b647-cb9ed24678a4@mon.a'

[7633905]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:10:16
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633905/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633905/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca8d744db342422b9ea863670d0f6ab0

    Command failed on smithi079 with status 5: 'sudo systemctl stop
    ceph-3c39a354-f039-11ee-b647-cb9ed24678a4@mon.a'

[7633906]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:08:17
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633906/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633906/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e3d539bc9e746788cdf6edf7b6cd576

    Command failed on smithi012 with status 5: 'sudo systemctl stop
    ceph-40f8eb92-f03a-11ee-b647-cb9ed24678a4@mon.a'

[7633907]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:12
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633907/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c3f911c4acf84da29b3342fe906fc43a

    Command failed on smithi055 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c3faca88-f03a-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\''

[7633908]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:10:57
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633908/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633908/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0438fa13df84ed08f3e85c683253cab

    Command failed on smithi040 with status 5: 'sudo systemctl stop
    ceph-5d9eb910-f03b-11ee-b647-cb9ed24678a4@mon.a'

[7633909]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   02:21:02
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633909/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633909/

    valgrind error: Leak_PossiblyLost calloc __trans_list_add

[7633910]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:08:23
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633910/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633910/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a3823ead26149378b0b939fc0ae21f8

    Command failed on smithi071 with status 5: 'sudo systemctl stop
    ceph-0a05a1ec-f03b-11ee-b647-cb9ed24678a4@mon.a'

[7633913]  fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633913/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1ccfa069173d4a4e8153e0b964488e2f

    Command failed on smithi052 with status 1: 'sudo yum upgrade -y linux-
    firmware'

[7633914]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client}
-----------------------------------------------------------------
time:   00:12:26
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633914/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=939d1f229cbf48c293ef631bfc73a5d7

    Command failed (workunit test client/test.sh) on smithi142 with status 134:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/client/test.sh'

[7633915]  fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:10:45
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633915/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633915/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4217bd48c4c14306a24a0909d39b6879

    Command failed on smithi175 with status 5: 'sudo systemctl stop
    ceph-9f5484b6-f03b-11ee-b647-cb9ed24678a4@mon.a'

[7633916]  fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:45:37
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633916/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b730b6870454425a9ee1a10d614264a7

    Test failure: test_reading_conf
    (tasks.cephfs.test_cephfs_shell.TestShellOpts)

[7633917]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:35:23
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633917/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f00ff40ea2be4838a63734e71e50d165

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
    (tasks.cephfs.test_mirroring.TestMirroring)

[7633918]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/openfiletable}
-----------------------------------------------------------------
time:   00:14:42
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633918/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0196f2ee4499456a9344a714c5a80b86

    Test failure: test_max_items_per_obj
    (tasks.cephfs.test_openfiletable.OpenFileTable)

[7633919]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:08:47
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633919/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4e0ae6603b04e9aa91f8ca512aae695

    Command failed on smithi164 with status 5: 'sudo systemctl stop
    ceph-5fb2d808-f03b-11ee-b647-cb9ed24678a4@mon.a'

[7633920]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/pool-perm}
-----------------------------------------------------------------
time:   00:15:41
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633920/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a4b189c9d6f1428eae33b410d42f1d71

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[7633922]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:12:03
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633922/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e6c744a131ef435f888bd892a95fffdf

    Command failed on smithi038 with status 5: 'sudo systemctl stop
    ceph-9c4a2b94-f03c-11ee-b647-cb9ed24678a4@mon.a'

[7633923]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:08:42
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633923/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633923/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3993c70867a441b58faade56c6f4e485

    Command failed on smithi019 with status 5: 'sudo systemctl stop
    ceph-6ae2120a-f03d-11ee-b647-cb9ed24678a4@mon.a'

[7633924]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:10:19
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633924/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6533e4a6a91e457a824683b6d870022f

    Command failed on smithi044 with status 5: 'sudo systemctl stop
    ceph-9a7b89a6-f03d-11ee-b647-cb9ed24678a4@mon.a'

[7633925]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:55
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633925/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a1328e88b114ddb879ce04c1485f9d2

    Command failed on smithi079 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f611fe12-f03d-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633926]  fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:14:53
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633926/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633926/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3a3db82b145a427c9485560efa67bbfa

    Command failed (workunit test suites/pjd.sh) on smithi121 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7633928]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:08:21
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633928/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e9295aa0c2794f268bfc61b6d229bcf6

    Command failed on smithi052 with status 5: 'sudo systemctl stop
    ceph-3eed8652-f03d-11ee-b647-cb9ed24678a4@mon.a'

[7633929]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:10:15
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633929/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633929/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ecf82a4ba6e347c2825d0fc4cb71fc40

    Command failed on smithi047 with status 5: 'sudo systemctl stop
    ceph-b09f704e-f03d-11ee-b647-cb9ed24678a4@mon.a'

[7633931]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:23
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633931/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633931/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cdc549df7364c2ba02eb5131610a4cd

    Command failed on smithi092 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d2621f10-f03d-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633932]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:08:54
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633932/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633932/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f018481aabb847bab142cebe1566af22

    Command failed on smithi026 with status 5: 'sudo systemctl stop
    ceph-5d40a614-f03f-11ee-b647-cb9ed24678a4@mon.a'

[7633934]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:09:53
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633934/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633934/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9dfd85240a074f5a8737eaf11c8ffe9d

    Command failed on smithi040 with status 5: 'sudo systemctl stop
    ceph-d453f13e-f03f-11ee-b647-cb9ed24678a4@mon.a'

[7633935]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:09:20
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633935/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633935/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6542ef282cd349d9b8baf5289bef13b5

    Command failed on smithi072 with status 5: 'sudo systemctl stop
    ceph-944dca38-f03f-11ee-b647-cb9ed24678a4@mon.a'

[7633936]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:10:51
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633936/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0484edb3d1048e2bf9fa80065b97f2a

    Command failed on smithi038 with status 5: 'sudo systemctl stop
    ceph-03590370-f040-11ee-b647-cb9ed24678a4@mon.a'

[7633937]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:01
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633937/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633937/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c08115315f54905afaa60ac43df69ef

    Command failed on smithi178 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0f2f4be6-f040-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633938]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:09:01
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633938/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633938/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5b231e66f9cd469a9b761e74ecbd4341

    Command failed on smithi029 with status 5: 'sudo systemctl stop
    ceph-6b9799b6-f03f-11ee-b647-cb9ed24678a4@mon.a'

[7633940]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:09:41
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633940/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633940/
sentry: https://sentry.ceph.com/organizations/ceph/?query=028880196078415292222f599d22a2ef

    Command failed on smithi081 with status 5: 'sudo systemctl stop
    ceph-c7580ec4-f040-11ee-b647-cb9ed24678a4@mon.a'

[7633941]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/xfstests-dev}
-----------------------------------------------------------------
time:   07:17:22
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633941/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633941/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a4f18884f3742c3802ac7629311f0cc

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7633942]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:58:45
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633942/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633942/

    "2024-04-01T16:20:25.282256+0000 mon.a (mon.0) 410 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7633943]  fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:17:45
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633943/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633943/
sentry: https://sentry.ceph.com/organizations/ceph/?query=651c4a2ebb714a3f86d27ac553a310f6

    Command failed (workunit test suites/pjd.sh) on smithi044 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7633944]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:12:33
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633944/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633944/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3171fa44ba234dae9c20fced10eda2f5

    Command failed on smithi060 with status 5: 'sudo systemctl stop
    ceph-1863769a-f042-11ee-b647-cb9ed24678a4@mon.a'

[7633945]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:30:44
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633945/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dadb0febb81c4e8db6a69e3baaed1636

    Command failed on smithi052 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    693dc6c4-f042-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633946]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:13:29
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633946/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633946/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9c76feabe7a46bdbc398baa93a060db

    Command failed on smithi047 with status 5: 'sudo systemctl stop
    ceph-68297cec-f042-11ee-b647-cb9ed24678a4@mon.a'

[7633948]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:13:12
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633948/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633948/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a023ed364d6d4e6ebd8c0f609ec6b1a9

    Command failed on smithi028 with status 5: 'sudo systemctl stop
    ceph-739030f8-f042-11ee-b647-cb9ed24678a4@mon.a'

[7633950]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:12:51
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633950/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633950/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e639320f4c04f51a8e905973ff95cdf

    Command failed on smithi073 with status 5: 'sudo systemctl stop
    ceph-c5e5151c-f042-11ee-b647-cb9ed24678a4@mon.a'

[7633951]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:27:40
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633951/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e0adfd94266432fa1ae43cdb447412d

    Command failed on smithi112 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    cb6a23ce-f042-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\''

[7633952]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:08:32
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633952/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633952/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d132186d5fa04e0b917e1619bc54607e

    Command failed on smithi003 with status 5: 'sudo systemctl stop
    ceph-a9410f32-f043-11ee-b647-cb9ed24678a4@mon.a'

[7633953]  fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
-----------------------------------------------------------------
time:   00:11:36
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633953/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633953/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30701853a375456595d7046918f26993

    Command failed on smithi173 with status 5: 'sudo systemctl stop
    ceph-8f2488d0-f044-11ee-b647-cb9ed24678a4@mon.smithi173'

[7633954]  fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:15:34
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633954/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633954/
sentry: https://sentry.ceph.com/organizations/ceph/?query=692f91e91cd742ad836ed0167c28d650

    Command failed (workunit test suites/pjd.sh) on smithi079 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d2271579d5e735cfd6d436b8349c004f06527cb0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7633955]  fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:13:32
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633955/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633955/
sentry: https://sentry.ceph.com/organizations/ceph/?query=383e5de59f5c46769b9d8cda1aa07270

    Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)

[7633956]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:10:49
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633956/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633956/
sentry: https://sentry.ceph.com/organizations/ceph/?query=267463dbad7f4d0686e93d1149495759

    Command failed on smithi081 with status 5: 'sudo systemctl stop
    ceph-774863ee-f044-11ee-b647-cb9ed24678a4@mon.a'

[7633958]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:52:08
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633958/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633958/

    valgrind error: Leak_StillReachable malloc malloc strdup

[7633959]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:08:41
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633959/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633959/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4390f58c89734b85a323a9c43d1fb18a

    Command failed on smithi072 with status 5: 'sudo systemctl stop
    ceph-d12153ae-f043-11ee-b647-cb9ed24678a4@mon.a'

[7633960]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:27:10
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633960/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633960/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ada42e93e54a4d8d905e1fe2c450e877

    Command failed on smithi038 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3de72798-f044-11ee-b647-cb9ed24678a4 -e
    sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 -- bash -c 'ceph orch ps'"

[7633961]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:10:58
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633961/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633961/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd78f5e065054f8e81bdd1a797efdd06

    Command failed on smithi029 with status 5: 'sudo systemctl stop
    ceph-838a9dac-f044-11ee-b647-cb9ed24678a4@mon.a'

[7633962]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:08:34
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633962/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633962/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d19a5b744f4e4548ac4f22a4f9aac55c

    Command failed on smithi055 with status 5: 'sudo systemctl stop ceph-
    adcbb444-f043-11ee-b647-cb9ed24678a4@mon.a'

[7633964]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:11:19
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633964/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633964/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36e3544060ca4e848ebaec64dec6db46

    Command failed on smithi094 with status 5: 'sudo systemctl stop ceph-
    cb0cb69c-f044-11ee-b647-cb9ed24678a4@mon.a'



Dead
=================================================================
[7633901]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633901/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633901/

    Error reimaging machines: Failed to power on smithi143

[7633911]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633911/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633911/
sentry: https://sentry.ceph.com/organizations/ceph/?query=758086dfe4df41719ba56eb19e016b92

    SSH connection to smithi163 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo
    kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O
    /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1d
    f0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/
    --input-file=-'

[7633912]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633912/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633912/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee0ba3010ae4c49a88c063ce4f79184

    SSH connection to smithi052 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo
    kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O
    /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1d
    f0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/
    --input-file=-'

[7633947]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633947/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633947/

    hit max job timeout



Pass
=================================================================
[7633894] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:22:50
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633894/

[7633903] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
time:   00:57:39
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633903/

[7633921] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:28:34
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633921/

[7633927] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:23:20
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633927/

[7633930] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:39:54
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633930/

[7633933] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:23:05
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633933/

[7633939] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest}
time:   00:25:16
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633939/

[7633949] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-dbench}}
time:   01:25:11
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633949/

[7633957] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:29:37
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633957/

[7633963] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:22:48
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633963/

[7633965] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage}
time:   00:21:07
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633965/

[7633966] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:22:44
info:   https://pulpito.ceph.com/vshankar-2024-04-01_14:09:42-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7633966/