Found 178 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474877/teuthology.log Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f2d29f6f0b6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Not-understood crash indication 2023-12-02T05:39:29.048 INFO:tasks.qemu.client.0.smithi151.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Found 6 distinct failure reasons Failure: Command failed on smithi142 with status 1: 'sudo yum -y install ceph' 2 jobs: ['7474855', '7474924'] suites intersection: ['openstack}'] suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'clusters/{fixed-3', 'conf/{disable-pool-app}', 'conf/{disable-pool-app}}', 'objectstore/bluestore-bitmap', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'rbd/pwl-cache/tmpfs/{1-base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rebuild_object_map}'] Crash: Command failed on smithi164 with status 139: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'echo \'"\'"\'{"type":"qcow","stream":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}\'"\'"\' | rbd migration prepare --import-only --source-spec-path - client.0.0\'' ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7656054db0] 2: /lib64/librbd.so.1(+0x2f5e91) [0x7f76574f5e91] 3: /lib64/librados.so.2(+0x12f5ae) [0x7f7657acc5ae] 4: /lib64/librados.so.2(+0xcc0df) [0x7f7657a690df] 5: /lib64/libstdc++.so.6(+0xdb924) [0x7f76564db924] 6: /lib64/libc.so.6(+0x9f802) [0x7f765609f802] 7: /lib64/libc.so.6(+0x3f450) [0x7f765603f450] 3 jobs: ['7475100', '7475013', '7474894'] suites intersection: ['2-clusters/{fixed-3', '4-supported-random-distro$/{centos_latest}', '5-data-pool/replicated', '6-prepare/qcow2-http', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/migration/{1-base/install'] suites union: ['2-clusters/{fixed-3', '3-objectstore/bluestore-bitmap', '3-objectstore/bluestore-comp-zlib', '3-objectstore/bluestore-hybrid', '4-supported-random-distro$/{centos_latest}', '5-data-pool/replicated', '6-prepare/qcow2-http', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/migration/{1-base/install'] Dead: b'2023-12-02T14:40:50.968 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7474817'] suites: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/replicated', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/qemu/{cache/writethrough', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_xfstests}'] Dead: None 1 jobs: ['7474877'] Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2d29f6eef8] 1 jobs: ['7475081'] suites: ['2-clusters/{fixed-3', '3-objectstore/bluestore-comp-snappy', '4-supported-random-distro$/{centos_latest}', '5-data-pool/none', '6-prepare/qcow2-file', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/migration/{1-base/install'] Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi057 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f60a9a54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f60a9aa154c] 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5611bd197f3c] 6: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 7: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 8: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 9: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 1 jobs: ['7474803'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}']