2023-11-28T16:16:53.904 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T16:16:53.904 INFO:teuthology.results:subset: '854/1024' 2023-11-28T16:16:53.904 INFO:teuthology.results:seed: '1098' 2023-11-28T18:34:20.486 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T18:34:20.510 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-29T06:20:18.730 INFO:teuthology.results:Tests finished! gathering results... 2023-11-29T06:20:37.132 INFO:scrape:Found 203 jobs 2023-11-29T06:20:44.803 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/teuthology.log 2023-11-29T06:20:44.804 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/teuthology.log 2023-11-29T06:20:44.804 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/teuthology.log 2023-11-29T06:20:44.804 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/teuthology.log 2023-11-29T06:20:46.074 WARNING:scrape:Misunderstood line: 2023-11-29T06:21:02.467 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469436/teuthology.log 2023-11-29T06:21:02.467 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469436/teuthology.log 2023-11-29T06:21:02.468 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469436/teuthology.log 2023-11-29T06:21:13.731 WARNING:scrape:Misunderstood line: 2023-11-29T06:21:17.606 WARNING:scrape:Misunderstood line: 2023-11-29T06:21:23.492 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469450/teuthology.log 2023-11-29T06:21:23.493 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469450/teuthology.log 2023-11-29T06:21:23.493 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469450/teuthology.log 2023-11-29T06:21:52.296 INFO:scrape:Found 33 distinct failure reasons 2023-11-29T06:21:52.297 INFO:scrape:Failure: Command failed on smithi178 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2023-11-29T06:21:52.297 INFO:scrape:52 jobs 2023-11-29T06:21:52.298 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-11-29T06:21:52.298 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/admin}', 'tasks/auto-repair}', 'tasks/damage}', 'tasks/forward-scrub}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mds-flush}', 'tasks/mds_creation_retry}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/recovery-fs}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/volumes/{overrides', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/snaps}', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/basic}}', 'test/clone}}', 'test/misc}}', 'thrashosds-health}', 'ubuntu_latest}}', 'wsync/no}', 'wsync/yes}'] 2023-11-29T06:21:52.298 INFO:scrape: 2023-11-29T06:21:52.298 INFO:scrape:Failure: failed to install new kernel version within timeout 2023-11-29T06:21:52.298 INFO:scrape:2 jobs: ['7469471', '7469509'] 2023-11-29T06:21:52.299 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health'] 2023-11-29T06:21:52.299 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}', 'tasks/openfiletable}'] 2023-11-29T06:21:52.299 INFO:scrape: 2023-11-29T06:21:52.299 INFO:scrape:Dead: b'2023-11-29T06:16:00.147 DEBUG:teuthology.exit:Finished running handlers' 2023-11-29T06:21:52.299 INFO:scrape:3 jobs: ['7469481', '7469376', '7469475'] 2023-11-29T06:21:52.299 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse'] 2023-11-29T06:21:52.299 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'tasks/failover}', 'tasks/mirror}}'] 2023-11-29T06:21:52.299 INFO:scrape: 2023-11-29T06:21:52.300 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-29T06:21:52.300 INFO:scrape:1 jobs: ['7469367'] 2023-11-29T06:21:52.300 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-11-29T06:21:52.300 INFO:scrape: 2023-11-29T06:21:52.300 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb77456ef52] 2023-11-29T06:21:52.300 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb773c54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb773ca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb77456ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb77456f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x560760b8e329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x560760b835b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x560760b3285b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x560760a6c521] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560760a73508] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x560760a68edd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x560760a697bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560760a55467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5607609d948b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607609d997c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5607609c34b5] 18: (DispatchQueue::entry()+0x53a) [0x7fb77476040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fb7747f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7fb773c9f802] 21: /lib64/libc.so.6(+0x3f450) [0x7fb773c3f450] 2023-11-29T06:21:52.300 INFO:scrape:1 jobs: ['7469337'] 2023-11-29T06:21:52.300 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-11-29T06:21:52.300 INFO:scrape: 2023-11-29T06:21:52.300 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 2023-11-29T06:21:52.301 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fa66d854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fa66d8a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 18: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 21: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-29T06:21:52.301 INFO:scrape:1 jobs: ['7469360'] 2023-11-29T06:21:52.301 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/sessionmap}'] 2023-11-29T06:21:52.301 INFO:scrape: 2023-11-29T06:21:52.301 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.118 2023-11-29T06:21:52.301 INFO:scrape:1 jobs: ['7469437'] 2023-11-29T06:21:52.301 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-11-29T06:21:52.301 INFO:scrape: 2023-11-29T06:21:52.301 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff875b6fef0] 2023-11-29T06:21:52.301 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff875242520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff875b6fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff875b7004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x563de34b83e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x563de34ad174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563de345e033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563de3398dec] 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x563de33ba590] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563de3395a91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563de339634c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563de3382837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563de33065fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563de3306acc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563de32f18e5] 18: (DispatchQueue::entry()+0x5ea) [0x7ff875d5c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff875dee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff875294b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff875326a00] 2023-11-29T06:21:52.302 INFO:scrape:1 jobs: ['7469389'] 2023-11-29T06:21:52.302 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-11-29T06:21:52.302 INFO:scrape: 2023-11-29T06:21:52.302 INFO:scrape:Dead: None 2023-11-29T06:21:52.302 INFO:scrape:3 jobs: ['7469492', '7469436', '7469450'] 2023-11-29T06:21:52.302 INFO:scrape: 2023-11-29T06:21:52.302 INFO:scrape:Failure: Command failed on smithi195 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:47c1c547372ed5bff7984647dc200bac0e019caf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8cfbdf8-8e12-11ee-95a2-87774f69a715 -- ceph mon dump -f json' 2023-11-29T06:21:52.302 INFO:scrape:4 jobs: ['7469373', '7469476', '7469429', '7469469'] 2023-11-29T06:21:52.302 INFO:scrape:suites intersection: [] 2023-11-29T06:21:52.303 INFO:scrape:suites union: ['1-mount', '1-ranks/1', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-workload/dbench', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'distro/single-container-host', 'distro/single-container-host}', 'fail_fs/no', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy'] 2023-11-29T06:21:52.303 INFO:scrape: 2023-11-29T06:21:52.303 INFO:scrape:Valgrind: mon (Leak_StillReachable) 2023-11-29T06:21:52.303 INFO:scrape:1 jobs: ['7469508'] 2023-11-29T06:21:52.303 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2023-11-29T06:21:52.303 INFO:scrape: 2023-11-29T06:21:52.303 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3e7af6fef0] 2023-11-29T06:21:52.303 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3e7a642520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3e7af6fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3e7af7004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x56334a7253e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56334a71a174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x56334a6cb033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x56334a605dec] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x56334a60cbe8] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x56334a602a91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x56334a60334c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56334a5ef837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56334a5735fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56334a573acc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56334a55e8e5] 18: (DispatchQueue::entry()+0x5ea) [0x7f3e7b15c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3e7b1ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3e7a694b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f3e7a726a00] 2023-11-29T06:21:52.303 INFO:scrape:1 jobs: ['7469515'] 2023-11-29T06:21:52.303 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-11-29T06:21:52.304 INFO:scrape: 2023-11-29T06:21:52.304 INFO:scrape:Failure: SSH connection to smithi118 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' 2023-11-29T06:21:52.304 INFO:scrape:6 jobs: ['7469503', '7469424', '7469474', '7469425', '7469458', '7469512'] 2023-11-29T06:21:52.304 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2023-11-29T06:21:52.304 INFO:scrape:suites union: ['1-ceph', '1-client', '2-logrotate}', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1-mds-3-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'multimds/no', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/ior-shared-file}', 'tasks/libcephfs_python}', 'tasks/{0-octopus'] 2023-11-29T06:21:52.304 INFO:scrape: 2023-11-29T06:21:52.304 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fdd0776fef0] 2023-11-29T06:21:52.304 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdd06e42520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fdd0776fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fdd0777004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x55f56d7cc3e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x55f56d7c1174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x55f56d772033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x55f56d6acdec] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f56d6b3be8] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x55f56d6a9a91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x55f56d6aa34c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f56d696837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x55f56d61a5fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f56d61aacc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f56d6058e5] 18: (DispatchQueue::entry()+0x5ea) [0x7fdd0795c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fdd079ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd06e94b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd06f26a00] 2023-11-29T06:21:52.304 INFO:scrape:1 jobs: ['7469456'] 2023-11-29T06:21:52.305 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-11-29T06:21:52.305 INFO:scrape: 2023-11-29T06:21:52.305 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 2023-11-29T06:21:52.305 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4e97642520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4e97f6fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f4e97f7004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x561ca0cb03e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x561ca0ca5174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x561ca0c56033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x561ca0b90dec] 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x561ca0bb2590] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x561ca0b8da91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x561ca0b8e34c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561ca0b7a837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x561ca0afe5fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561ca0afeacc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561ca0ae98e5] 18: (DispatchQueue::entry()+0x5ea) [0x7f4e9815c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f4e981ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4e97694b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4e97726a00] 2023-11-29T06:21:52.305 INFO:scrape:1 jobs: ['7469457'] 2023-11-29T06:21:52.305 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-11-29T06:21:52.305 INFO:scrape: 2023-11-29T06:21:52.305 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fce94a81ef0] 2023-11-29T06:21:52.305 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fce943bc520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fce94a81ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fce94a8204a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x55edd1d883e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x55edd1d7d174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x55edd1d2e033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x55edd1c68dec] 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x55edd1c8a590] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x55edd1c65a91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x55edd1c6634c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55edd1c52837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x55edd1bd65fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55edd1bd6acc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55edd1bc18e5] 18: (DispatchQueue::entry()+0x5ea) [0x7fce94c6e11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fce94d00991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fce9440eb43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fce944a0a00] 2023-11-29T06:21:52.305 INFO:scrape:1 jobs: ['7469328'] 2023-11-29T06:21:52.306 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-11-29T06:21:52.306 INFO:scrape: 2023-11-29T06:21:52.306 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f981236fef0] 2023-11-29T06:21:52.306 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f9811a42520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f981236fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f981237004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x56399de503e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56399de45174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x56399ddf6033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x56399dd30dec] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x56399dd37be8] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x56399dd2da91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x56399dd2e34c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56399dd1a837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56399dc9e5fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56399dc9eacc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56399dc898e5] 18: (DispatchQueue::entry()+0x5ea) [0x7f981255c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f98125ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f9811a94b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9811b26a00] 2023-11-29T06:21:52.306 INFO:scrape:1 jobs: ['7469520'] 2023-11-29T06:21:52.306 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-11-29T06:21:52.306 INFO:scrape: 2023-11-29T06:21:52.306 INFO:scrape:Failure: Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2023-11-29T06:21:52.306 INFO:scrape:1 jobs: ['7469482'] 2023-11-29T06:21:52.306 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-11-29T06:21:52.307 INFO:scrape: 2023-11-29T06:21:52.307 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2023-11-29T06:21:52.307 INFO:scrape:1 jobs: ['7469510'] 2023-11-29T06:21:52.307 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-11-29T06:21:52.307 INFO:scrape: 2023-11-29T06:21:52.307 INFO:scrape:Failure: Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree) 2023-11-29T06:21:52.307 INFO:scrape:1 jobs: ['7469421'] 2023-11-29T06:21:52.307 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/asok_dump_tree}'] 2023-11-29T06:21:52.307 INFO:scrape: 2023-11-29T06:21:52.307 INFO:scrape:Failure: '2023-11-28T18:06:46.435175+00:00 smithi120 kernel: [ 2113.973541] BUG: kernel NULL pointer dereference, address: 0000000000000610 ' in syslog 2023-11-29T06:21:52.307 INFO:scrape:1 jobs: ['7469404'] 2023-11-29T06:21:52.308 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'ubuntu_latest}}'] 2023-11-29T06:21:52.308 INFO:scrape: 2023-11-29T06:21:52.308 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc5876ef52] 2023-11-29T06:21:52.308 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fcc57e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fcc57ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcc5876ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fcc5876f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x555924c6b329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x555924c605b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x555924c0f85b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x555924b49521] 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x555924b6b070] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x555924b45edd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x555924b467bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555924b32467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x555924ab648b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555924ab697c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x555924aa04b5] 18: (DispatchQueue::entry()+0x53a) [0x7fcc5896040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fcc589f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7fcc57e9f802] 21: /lib64/libc.so.6(+0x3f450) [0x7fcc57e3f450] 2023-11-29T06:21:52.308 INFO:scrape:1 jobs: ['7469461'] 2023-11-29T06:21:52.308 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/lockdep}'] 2023-11-29T06:21:52.308 INFO:scrape: 2023-11-29T06:21:52.308 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6a7316fef0] 2023-11-29T06:21:52.308 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6a72842520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6a7316fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f6a7317004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x563d9224e3e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x563d92243174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563d921f4033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563d9212edec] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563d92135be8] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563d9212ba91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563d9212c34c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563d92118837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563d9209c5fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563d9209cacc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563d920878e5] 18: (DispatchQueue::entry()+0x5ea) [0x7f6a7335c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f6a733ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f6a72894b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f6a72926a00] 2023-11-29T06:21:52.308 INFO:scrape:1 jobs: ['7469486'] 2023-11-29T06:21:52.309 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2023-11-29T06:21:52.309 INFO:scrape: 2023-11-29T06:21:52.309 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fbfbe76fef0] 2023-11-29T06:21:52.309 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbfbde42520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fbfbe76fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fbfbe77004a] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x341) [0x5621d94e0d11] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x5621d9279579] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x5621d927b5fb] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5621d927bacc] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5621d92668e5] 12: (DispatchQueue::entry()+0x5ea) [0x7fbfbe95c11a] 13: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fbfbe9ee991] 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fbfbde94b43] 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fbfbdf26a00] 2023-11-29T06:21:52.309 INFO:scrape:1 jobs: ['7469379'] 2023-11-29T06:21:52.309 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-29T06:21:52.309 INFO:scrape: 2023-11-29T06:21:52.309 INFO:scrape:Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x4bd7e96] 2023-11-29T06:21:52.309 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x4bd7e96] 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x4bd804a] 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x4003e8] 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x3f5174] 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x3a6033] 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x2e0dec] 7: (Server::handle_client_mkdir(boost::intrusive_ptr&)+0x60) [0x2fc730] 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x2dda91] 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x2de34c] 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x2ca837] 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x24e5fb] 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x24eacc] 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x2398e5] 14: (DispatchQueue::entry()+0x5ea) [0x4dc411a] 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x4e56991] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x57a5b43] 2023-11-29T06:21:52.309 INFO:scrape:2 jobs: ['7469426', '7469347'] 2023-11-29T06:21:52.310 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2023-11-29T06:21:52.310 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2023-11-29T06:21:52.310 INFO:scrape: 2023-11-29T06:21:52.310 INFO:scrape:Failure: Command failed on smithi086 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e43b60d6-8e16-11ee-95a2-87774f69a715 -- ceph mon dump -f json' 2023-11-29T06:21:52.310 INFO:scrape:1 jobs: ['7469411'] 2023-11-29T06:21:52.310 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-11-29T06:21:52.310 INFO:scrape: 2023-11-29T06:21:52.310 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-11-29T06:21:52.310 INFO:scrape:1 jobs: ['7469397'] 2023-11-29T06:21:52.310 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-11-29T06:21:52.311 INFO:scrape: 2023-11-29T06:21:52.311 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1459f6ef52] 2023-11-29T06:21:52.311 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1459654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f14596a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1459f6ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1459f6f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x561cd19bb329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x561cd19b05b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x561cd195f85b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x561cd1899521] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x561cd18a0508] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x561cd1895edd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x561cd18967bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561cd1882467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561cd180648b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561cd180697c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x561cd17f04b5] 18: (DispatchQueue::entry()+0x53a) [0x7f145a16040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f145a1f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7f145969f802] 21: /lib64/libc.so.6(+0x3f450) [0x7f145963f450] 2023-11-29T06:21:52.311 INFO:scrape:1 jobs: ['7469448'] 2023-11-29T06:21:52.311 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-11-29T06:21:52.311 INFO:scrape: 2023-11-29T06:21:52.311 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 2023-11-29T06:21:52.311 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fe33b654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fe33b6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 18: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 21: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-29T06:21:52.311 INFO:scrape:1 jobs: ['7469398'] 2023-11-29T06:21:52.311 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-11-29T06:21:52.312 INFO:scrape: 2023-11-29T06:21:52.312 INFO:scrape:Failure: Test failure: test_replay_beacon_estimated_time (tasks.cephfs.test_failover.TestFailoverBeaconHealth) 2023-11-29T06:21:52.312 INFO:scrape:1 jobs: ['7469438'] 2023-11-29T06:21:52.312 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/failover}'] 2023-11-29T06:21:52.312 INFO:scrape: 2023-11-29T06:21:52.312 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb38276ef52] 2023-11-29T06:21:52.312 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb381e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb381ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb38276ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb38276f0b6] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5574f8ef3329] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5574f8ee85b5] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5574f8e9785b] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5574f8dd1521] 11: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x90) [0x5574f8df3070] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5574f8dcdedd] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5574f8dce7bc] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5574f8dba467] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5574f8d3e48b] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5574f8d3e97c] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5574f8d284b5] 18: (DispatchQueue::entry()+0x53a) [0x7fb38296040a] 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fb3829f24a1] 20: /lib64/libc.so.6(+0x9f802) [0x7fb381e9f802] 21: /lib64/libc.so.6(+0x3f450) [0x7fb381e3f450] 2023-11-29T06:21:52.312 INFO:scrape:1 jobs: ['7469331'] 2023-11-29T06:21:52.312 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2023-11-29T06:21:52.312 INFO:scrape: 2023-11-29T06:21:52.313 INFO:scrape:Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 2023-11-29T06:21:52.313 INFO:scrape:1 jobs: ['7469341'] 2023-11-29T06:21:52.313 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2023-11-29T06:21:52.313 INFO:scrape: 2023-11-29T06:21:52.313 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc50956fef0] 2023-11-29T06:21:52.313 INFO:scrape:ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc508c42520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc50956fef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fc50957004a] 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x55cda60c03e8] 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x55cda60b5174] 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x55cda6066033] 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x55cda5fa0dec] 11: (Server::handle_client_mkdir(boost::intrusive_ptr&)+0x60) [0x55cda5fbc730] 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x55cda5f9da91] 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x55cda5f9e34c] 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55cda5f8a837] 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x55cda5f0e5fb] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55cda5f0eacc] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55cda5ef98e5] 18: (DispatchQueue::entry()+0x5ea) [0x7fc50975c11a] 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fc5097ee991] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc508c94b43] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc508d26a00] 2023-11-29T06:21:52.313 INFO:scrape:1 jobs: ['7469459'] 2023-11-29T06:21:52.313 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/exports}'] 2023-11-29T06:21:52.313 INFO:scrape: 2023-11-29T06:21:52.324 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/ failed: 87 dead: 11 running: 0 waiting: 0 queued: 0 passed: 105 Fail ================================================================= [7469319] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:29 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469319/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=72f82c04674c4d6caa00ea371080e40d Command failed on smithi087 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:09:22 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469326/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=657c5089f2444f6fbc83cdc798dc10f9 Command failed on smithi098 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469327] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:13:33 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469327/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b58717843064912a37584b683e514a6 Command failed on smithi188 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:51 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469328/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c8256c8550847189c6989cf0f6405b7 Command failed on smithi071 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7469331] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:28:50 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469331/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469331/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3426a22b9d2475cbc5a402903d74d40 Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics), test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [7469336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:14:20 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469336/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8d57eb220784a3bbe3269e0fb4b0808 Command failed on smithi072 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469337] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:28:42 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469337/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69d740ae532541478c973822b7faa668 Command failed (workunit test suites/ffsb.sh) on smithi107 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7469338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:09:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469338/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcb87f5049734d9b8e0c1dc80a695fc1 Command failed on smithi116 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469341] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:17:04 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469341/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34a568bd8c9749f496f58e1d56e91ac8 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) [7469343] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469343/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4008b10810474da0a445667b5870c1fe Command failed on smithi087 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469345] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:09:40 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469345/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abb1f23ae4b2498fb81409b11f47dc1c Command failed on smithi162 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469347] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:30:31 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469347/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a7b6fb1f8984388b11e4a5bbc14b4c8 Command failed (workunit test suites/fsstress.sh) on smithi039 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7469348] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:14:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469348/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88338e9bd1cb4972828af1cea5e8c327 Command failed on smithi070 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469352] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:54 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469352/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4da53d50c7034d9e80c978c15aaf148d Command failed on smithi138 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469353] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:10:04 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469353/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c4be41282ea4f22bd2767ef23948f33 Command failed on smithi112 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469356] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:10:20 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469356/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8eb76ebf98554174937224c233cb51da Command failed on smithi144 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469360] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:25:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22f534bcf0f146c795677ca44fc2d08b Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap), test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) [7469363] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469363/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4425fec03be247a6aac696869d37b9e5 Command failed on smithi087 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469364] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:30 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469364/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7c34b0e33524f129f624ffdbfcd8f50 Command failed on smithi192 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469367] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:52:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469367/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a34097b5c5584901ab8dae1bd77c4c9a Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7469368] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:13:08 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469368/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5db0e039318049968285c840acef20d1 Command failed on smithi162 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469373] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:18:14 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469373/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c2501e33595446194bdacce0666988c Command failed on smithi195 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:47c1c547372ed5bff7984647dc200bac0e019caf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8cfbdf8-8e12-11ee-95a2-87774f69a715 -- ceph mon dump -f json' [7469379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:37:51 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469379/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed38a3ac8c024d08911f53b0ad03203b Test failure: test_multimds_hardlink (tasks.cephfs.test_snapshots.TestSnapshots), test_multimds_hardlink (tasks.cephfs.test_snapshots.TestSnapshots) [7469380] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:21 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469380/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=553ec1d960094986a935e6d0bd34032a Command failed on smithi181 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469381] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:13:53 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469381/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57d946acdbee4ee1b4c7314c3068b45e Command failed on smithi192 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469383] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469383/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57fd485001b946d9a345edde5a1ff9c9 Command failed on smithi001 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469384] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:09:45 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469384/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469384/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9174b470e814387b839ffd4168764d5 Command failed on smithi178 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469389] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:58 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469389/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52ed555f9ed84005bc52fdd51da8a335 Command failed on smithi031 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7469390] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:13:43 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469390/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a6968e2a2afb42a1922c89a50212993a Command failed on smithi159 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469394] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:09:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469394/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36a116fef6f04124af4ace58a0200e27 Command failed on smithi184 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469397] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:50:41 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469397/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e505c41250949faa9497e2a782da9bd Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7469398] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:28:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=001eb7dd2d2c4a7db44673ab0344338c Command failed (workunit test suites/ffsb.sh) on smithi098 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7469401] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:55 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469401/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec5cbe9a41874c6f94bc05bebcec63e5 Command failed on smithi123 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469404] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:51:06 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469404/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469404/ '2023-11-28T18:06:46.435175+00:00 smithi120 kernel: [ 2113.973541] BUG: kernel NULL pointer dereference, address: 0000000000000610 ' in syslog [7469408] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:09:46 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469408/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f430202e0d84bdc97ef729c05090468 Command failed on smithi178 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469409] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:33 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469409/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56e276f1d8e04255a5f1f898ba57eefc Command failed on smithi167 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469411] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:17 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469411/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bdfd47b22f564430bbb152d1fcab44cb Command failed on smithi086 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e43b60d6-8e16-11ee-95a2-87774f69a715 -- ceph mon dump -f json' [7469412] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469412/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d5bc608dbfa4663aa91d5f0001a7dc0 Command failed on smithi184 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469413] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469413/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=308037975f014c42b709f98b072ae99f Command failed on smithi144 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469414] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:09:30 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469414/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f43f98d970604c0abb638353c72c174d Command failed on smithi104 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469418] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469418/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=639c7248a1d04d7aa72b45e28f71954a Command failed on smithi002 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:13:58 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469419/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469419/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59fd740fc87d49a2a5cab974e083f8b6 Command failed on smithi159 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469420] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:09:28 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469420/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d3c662e47134a5482ea4781b92f4d95 Command failed on smithi136 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469421] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:14:31 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469421/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469421/ sentry: https://sentry.ceph.com/organizations/ceph/?query=169497bb9b2f488f80ebc3cb611c99f5 Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree) [7469424] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469424/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469424/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c65e3408f344be6aea4249dcf0b3b8a Command failed on smithi118 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' [7469426] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:30:06 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469426/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3818dfda6294fb6900511aacc6046e1 Command failed (workunit test suites/fsstress.sh) on smithi029 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7469427] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:09:47 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469427/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6be57493efa14126836e44b7209349b4 Command failed on smithi203 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469429] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469429/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469429/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf9db99503eb4189b8026ecbfe895d93 Command failed on smithi178 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 59072844-8e19-11ee-95a2-87774f69a715 -- ceph mon dump -f json' [7469432] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:14:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469432/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469432/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11e1dc9a7ec24e25b438d4232ecddd2c Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469435] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:37 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469435/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469435/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2914a56049f34291910d4dea2b1f8d9a Command failed on smithi040 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469437] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469437/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469437/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a50a6d6c16634b9ab1136f75c26b8080 [Errno None] Unable to connect to port 22 on 172.21.15.118 [7469438] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:47:30 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469438/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=361bf4f328854f238555a3ded1b39ab3 Test failure: test_replay_beacon_estimated_time (tasks.cephfs.test_failover.TestFailoverBeaconHealth) [7469442] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:10:03 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469442/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469442/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d191cc94ff1347a9ada34a487aac4921 Command failed on smithi188 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469448] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:18 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469448/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469448/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae5c41054182487988f854bd742ecb45 Command failed on smithi133 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7469453] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:13:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469453/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7040224415174cc8a34ac87b512e915b Command failed on smithi125 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469454] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:10:01 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469454/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469454/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa980d31bf15405980116f50fd3361e7 Command failed on smithi045 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469456] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:39:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469456/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afb10c6922084df2a938e43d9f5137e6 Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan), test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) [7469457] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:47:32 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9d491af047e42f68657ad2fc302775b Command failed (workunit test suites/ffsb.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7469459] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:31:38 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469459/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d95c8cb01ac45edb20eeae91a06c2ae Test failure: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) [7469460] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:13:34 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469460/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=926a342ea2414c5685f4e53465a23cb8 Command failed on smithi113 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469461] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:23:02 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469461/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fde1a2dacd334fa9872d526f2257f8ac Command failed (workunit test suites/dbench.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7469462] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:01 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469462/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=078218520c6041c09186f5f0c15b06dc Command failed on smithi098 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469464] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:09:45 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469464/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7e1ede11490489da651a5601be79ca2 Command failed on smithi082 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469465] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:09:56 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469465/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cde4c2c8e6da4d12be6b8866c33976ec Command failed on smithi137 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469469] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:18:44 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469469/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d6f39562aaf472aaa2d3d9393f9cd72 Command failed on smithi119 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:47c1c547372ed5bff7984647dc200bac0e019caf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc3ae0a0-8e1b-11ee-95a2-87774f69a715 -- ceph mon dump -f json' [7469471] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469471/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bea81ce15fd842d4bf949c2d2f20e36e failed to install new kernel version within timeout [7469476] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:11:14 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469476/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6565f6dfefe64ddc92cc42803adc7583 Command failed on smithi099 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:47c1c547372ed5bff7984647dc200bac0e019caf pull' [7469482] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:06:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469482/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=988dd40ceac449a0b318ea12681b2f75 Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) [7469483] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:14:08 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469483/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=198032aa0afa4046a1d5188643047b58 Command failed on smithi094 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469486] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:29:24 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469486/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469486/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37686a0a69f34edabc34082d7f380747 Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair), test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) [7469487] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:09:47 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469487/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=444b4ef148404ad79f8ad16c5c6fa393 Command failed on smithi178 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469491] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:10:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469491/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13c1a37a24874529b326e21640261592 Command failed on smithi089 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469493] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:10:22 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469493/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40ba71ea703f4964baf77a3ab3ee338c Command failed on smithi084 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469499] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:13:19 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469499/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469499/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ce4f6c61f74485aa60de63046dbde9b Command failed on smithi155 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469500] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469500/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b63389fda2cd491bb8163eb2d58f6bf2 Command failed on smithi121 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469502] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:50 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469502/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469502/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e9feca3e04d4bc0bcf016af9c6e708a Command failed on smithi170 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469506] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:09:38 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469506/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b61926f8cb784a1da5112ea544ca2c77 Command failed on smithi120 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469507] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:09:35 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469507/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c402a10a35974b449acc36cd2dbb8905 Command failed on smithi176 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469508] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:39:44 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469508/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469508/ saw valgrind issues [7469509] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469509/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469509/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb54fb22ad184299aaec90c8689eb597 failed to install new kernel version within timeout [7469510] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:24:31 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469510/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f10087c5f654590bf6d82720aec331b Command failed (workunit test suites/fsstress.sh) on smithi192 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7469511] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:13:32 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469511/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35ce4978fc334d1898b9050209d63c7b Command failed on smithi092 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7469513] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:09:39 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469513/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab179149546f40dabfa54253ba1a0ca3 Command failed on smithi099 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469514] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:28 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469514/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469514/ sentry: https://sentry.ceph.com/organizations/ceph/?query=044c916caa6f493c997d570934f3f5f4 Command failed on smithi124 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:06 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469515/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469515/ sentry: https://sentry.ceph.com/organizations/ceph/?query=130453f9711f4b48a554bc001cab8a7d Command failed on smithi037 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7469516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:09:37 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469516/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5256331983584bc1bd56c0ee35700bcd Command failed on smithi119 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7469520] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:31:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469520/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6be28ad3a1034ee3a9604fd32c3f11ed Command failed (workunit test suites/ffsb.sh) on smithi089 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' Dead ================================================================= [7469376] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469376/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469376/ hit max job timeout [7469425] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469425/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469425/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc0360f1b322483db3bc6f267f0a648a SSH connection to smithi118 was lost: 'sudo dpkg -i /tmp/linux-image.deb' [7469436] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469436/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469436/ Error reimaging machines: Expected smithi118's OS to be rhel 8.6 but found ubuntu 22.04 [7469450] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469450/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469450/ Error reimaging machines: Expected smithi118's OS to be ubuntu 22.04 but found centos 8 [7469458] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469458/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469458/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fde0e72980e49c0b7eace43811853c0 SSH connection to smithi118 was lost: 'sudo yum upgrade -y linux-firmware' [7469474] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469474/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34f269da7b82466485663023b4a81df9 SSH connection to smithi118 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' [7469475] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469475/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469475/ hit max job timeout [7469481] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469481/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469481/ hit max job timeout [7469492] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469492/ Error reimaging machines: Expected smithi118's OS to be centos 8 but found ubuntu 22.04 [7469503] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469503/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90ef539a0314417bbefbaf095be34d3a SSH connection to smithi118 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' [7469512] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469512/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16c40ada7cab41118debf0211a99edfe SSH connection to smithi118 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' Pass ================================================================= [7469318] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:14:57 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469318/ [7469320] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:26 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469320/ [7469321] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:27:01 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469321/ [7469322] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:20:39 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469322/ [7469323] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:26:19 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469323/ [7469324] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:15:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469324/ [7469325] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:43 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469325/ [7469329] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:41 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469329/ [7469330] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:48:20 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469330/ [7469332] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:16 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469332/ [7469333] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:18:53 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469333/ [7469334] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:22 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469334/ [7469335] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:45 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469335/ [7469339] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 02:03:36 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469339/ [7469340] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:32:40 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469340/ [7469342] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:04:29 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469342/ [7469344] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:15:36 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469344/ [7469346] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:43 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469346/ [7469349] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:18:44 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469349/ [7469350] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469350/ [7469351] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469351/ [7469354] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469354/ [7469355] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:24 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469355/ [7469357] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:40:18 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469357/ [7469358] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:19 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469358/ [7469359] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469359/ [7469361] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:57:09 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469361/ [7469362] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:23:48 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469362/ [7469365] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:19:10 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469365/ [7469366] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:17:14 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469366/ [7469369] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:26 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469369/ [7469370] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:15:37 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469370/ [7469371] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 04:23:28 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469371/ [7469372] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:10 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469372/ [7469374] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:27:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469374/ [7469375] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:48 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469375/ [7469377] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:26 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469377/ [7469378] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:44:30 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469378/ [7469382] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:15 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469382/ [7469385] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:13 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469385/ [7469386] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:27:17 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469386/ [7469387] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:49:33 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469387/ [7469388] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:14:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469388/ [7469391] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:16:22 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469391/ [7469392] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:13 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469392/ [7469393] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:20:04 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469393/ [7469395] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:54 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469395/ [7469396] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:21:45 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469396/ [7469399] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:21:25 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469399/ [7469400] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:21 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469400/ [7469402] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:04 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469402/ [7469403] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:19:17 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469403/ [7469405] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:20:16 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469405/ [7469406] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469406/ [7469407] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:16:06 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469407/ [7469410] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:20:15 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469410/ [7469415] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:43 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469415/ [7469416] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:21 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469416/ [7469417] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:15:41 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469417/ [7469422] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:27:48 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469422/ [7469423] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:24 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469423/ [7469428] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:31 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469428/ [7469430] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:18 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469430/ [7469431] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469431/ [7469433] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:21:03 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469433/ [7469434] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:16:16 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469434/ [7469439] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:20:23 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469439/ [7469440] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:11 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469440/ [7469441] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:26:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469441/ [7469443] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 02:48:17 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469443/ [7469444] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:08 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469444/ [7469445] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:16:44 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469445/ [7469446] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:23:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469446/ [7469447] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:21 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469447/ [7469449] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:20 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469449/ [7469451] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469451/ [7469452] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 01:04:01 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469452/ [7469455] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:29 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469455/ [7469463] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:13 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469463/ [7469466] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:23:14 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469466/ [7469467] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:25:51 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469467/ [7469468] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:16:47 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469468/ [7469470] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:23:41 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469470/ [7469472] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror} time: 02:22:36 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469472/ [7469473] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} time: 00:37:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469473/ [7469477] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:17 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469477/ [7469478] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:52:21 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469478/ [7469479] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 02:31:19 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469479/ [7469480] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:17:18 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469480/ [7469484] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:27 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469484/ [7469485] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469485/ [7469488] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:52:51 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469488/ [7469489] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:18 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469489/ [7469490] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:52 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469490/ [7469494] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:25:30 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469494/ [7469495] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:49 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469495/ [7469496] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:01 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469496/ [7469497] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469497/ [7469498] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 01:19:07 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469498/ [7469501] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:05 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469501/ [7469504] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:07 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469504/ [7469505] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:36 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469505/ [7469517] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:22:12 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469517/ [7469518] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:18:43 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469518/ [7469519] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:16 info: https://pulpito.ceph.com/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469519/