Found 221 jobs Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7637051/teuthology.log Found 10 distinct failure reasons Failure: timed out 1 jobs: ['7636924'] suites: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/replicated', 'extra-conf/copy-on-read', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/librbd/{cache/none', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_defaults}'] Failure: Failed to reconnect to smithi184 1 jobs: ['7636941'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/device/{base/install', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_fsx_nbd}'] Failure: Command failed (workunit test rbd/diff_continuous.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d7e176b41636f150fe82aaedea82def3e9af0203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=krbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' 1 jobs: ['7637002'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/device/{base/install', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/diff-continuous-krbd}'] Failure: Command failed on smithi028 with status 5: 'sudo systemctl stop ceph-50b90b6c-f5d9-11ee-b649-cb9ed24678a4@mon.a' 2 jobs: ['7636870', '7636865'] suites intersection: ['conf/{disable-pool-app}', 'openstack}'] suites union: ['base/install', 'centos_latest', 'cluster/{fixed-3', 'cluster/{fixed-4', 'conf/{disable-pool-app}', 'openstack}', 'rbd/iscsi/{0-single-container-host', 'rbd/nvmeof/{base/install', 'workloads/cephadm_iscsi}', 'workloads/nvmeof_initiator}'] Failure: reached maximum tries (51) after waiting for 300 seconds 1 jobs: ['7636881'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] Dead: None 2 jobs: ['7637051', '7636923'] Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x4cd0f3a] ceph version 19.0.0-2578-ge5c885fe (e5c885fe95ee982ff7b45214daa61b1a6d09d57d) squid (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x4cd0f3a] 2: /usr/lib64/ceph/libceph-common.so.2(+0x1810f8) [0x4cd10f8] 3: (rbd::mirror::LeaderWatcher::cancel_timer_task()+0x14c) [0x54789c] 4: (rbd::mirror::LeaderWatcher::schedule_timer_task(std::__cxx11::basic_string, std::allocator > const&, int, bool, void (rbd::mirror::LeaderWatcher::*)(), bool)+0x78) [0x547928] 5: (rbd::mirror::LeaderWatcher::schedule_acquire_leader_lock(unsigned int)+0xa3) [0x5498e3] 6: (rbd::mirror::LeaderWatcher::handle_get_locker(int, librbd::managed_lock::Locker&)+0x4bd) [0x54a89d] 7: ceph_test_rbd_mirror(+0x3aa0fd) [0x4b20fd] 8: (boost::asio::detail::executor_op, std::allocator, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xa0) [0x524580] 9: ceph_test_rbd_mirror(+0x41cb96) [0x524b96] 10: (boost::asio::detail::executor_op, 0ul> const, void>, boost::asio::detail::recycling_allocator, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0xcc) [0x524f6c] 11: /lib64/librados.so.2(+0x13671e) [0x499571e] 12: /lib64/librados.so.2(+0xd0a1f) [0x492fa1f] 13: /lib64/libstdc++.so.6(+0xdb924) [0x5a14924] 14: /lib64/libc.so.6(+0x9f802) [0x5cf5802] 1 jobs: ['7636914'] suites: ['centos_latest', 'clusters/{fixed-1', 'conf/{disable-pool-app}', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/rbd_mirror}'] Failure: "2024-04-08T18:12:45.956312+0000 mon.a (mon.0) 188 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 1 jobs: ['7636855'] suites: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-workunit-journaling}'] Failure: Command failed (workunit test rbd/rbd_mirror.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d7e176b41636f150fe82aaedea82def3e9af0203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh' 1 jobs: ['7637020'] suites: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}'] Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d7e176b41636f150fe82aaedea82def3e9af0203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 1 jobs: ['7637007'] suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/ssd', '6-cache-size/8G', '7-workloads/c_api_tests_with_defaults', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/home/{1-base/install']