Found 4 jobs Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] Found 4 distinct failure reasons Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 1 jobs: ['7233031'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] Failure: "2023-04-05T20:08:21.169538+0000 mon.a (mon.0) 516 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 1 jobs: ['7233029'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 1 jobs: ['7233030'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/alternate-pool}'] Crash: Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=44b6581f04c8d41f47aa0216a156fa52bb6cba23 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh' ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5582e5f420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f5583381db0] 5: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 6: (Monitor::~Monitor()+0x146b) [0x5632a7832d1b] 7: (Monitor::~Monitor()+0xd) [0x5632a7832d9d] 8: main() 9: __libc_start_main() 10: _start() 1 jobs: ['7233028'] suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/fs/misc}', 'wsync/{no}}']