Found 343 jobs Found 6 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/cephtool/test.sh 1 jobs: ['5957697'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Dead: b'2021-03-12T17:30:52.618 DEBUG:teuthology.exit:Finished running handlers' 2 jobs: ['5957589', '5957804'] suites intersection: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] suites union: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802fb56f38ae88bca7c1d11749d764133a3af997 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' 4 jobs: ['5957618', '5957876', '5957703', '5957795'] suites intersection: ['task/test_e2e}'] suites union: ['rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}'] Crash: Found coredumps on ubuntu@smithi104.front.sepia.ceph.com ceph version 16.1.0-730-g802fb56f (802fb56f38ae88bca7c1d11749d764133a3af997) pacific (rc) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7fe8a9c2bdc0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9006b) [0x7fe8a924e06b] 5: /lib64/libstdc++.so.6(+0x9650c) [0x7fe8a925450c] 6: /lib64/libstdc++.so.6(+0x96567) [0x7fe8a9254567] 7: /lib64/libstdc++.so.6(+0x967c8) [0x7fe8a92547c8] 8: ceph-osd(+0x5890ee) [0x55b0e90ce0ee] 9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned int)+0x26) [0x55b0e99e2686] 10: (ceph::buffer::v15_2_0::create(unsigned int)+0x26) [0x55b0e99e26d6] 11: (ceph::buffer::v15_2_0::list::obtain_contiguous_space(unsigned int)+0xab) [0x55b0e99e28db] 12: (ECSubReadReply::encode(ceph::buffer::v15_2_0::list&) const+0xfd) [0x55b0e9a7e26d] 13: (MOSDECSubOpReadReply::encode_payload(unsigned long)+0x65) [0x55b0e957d7c5] 14: (Message::encode(unsigned long, int, bool)+0x2e) [0x55b0e994be5e] 15: (ProtocolV2::prepare_send_message(unsigned long, Message*)+0x44) [0x55b0e9b9f774] 16: (ProtocolV2::send_message(Message*)+0x3ae) [0x55b0e9b9fdde] 17: (AsyncConnection::send_message(Message*)+0x49e) [0x55b0e9b7dc2e] 18: (ECBackend::_handle_message(boost::intrusive_ptr)+0x4d9) [0x55b0e957bee9] 19: (PGBackend::handle_message(boost::intrusive_ptr)+0x52) [0x55b0e93845f2] 20: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5de) [0x55b0e93278de] 21: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x309) [0x55b0e91b2279] 22: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x68) [0x55b0e940e318] 23: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x55b0e91d16c8] 24: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b0e9838054] 25: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b0e983acf4] 26: /lib64/libpthread.so.0(+0x82de) [0x7fe8a9c212de] 27: clone() 1 jobs: ['5957630'] suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/classic', 'msgr-failures/fastclose', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-fast-read-overwrites}'] Failure: Command failed on smithi159 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:802fb56f38ae88bca7c1d11749d764133a3af997 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe169460-8308-11eb-9086-001a4aab830c -- ceph orch host ls --format=json' 2 jobs: ['5957863', '5957626'] suites intersection: [] suites union: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v1only', 'rados/cephadm/smoke-roleless/{distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04_kubic_testing', 'start', 'start}', 'tasks/rados_api_tests}'] Failure: Scrubbing terminated -- not all pgs were active and clean. 1 jobs: ['5957602'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/lost-unfound-delete', 'supported-random-distro$/{rhel_8}}']