Found 113 jobs Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2021-02-15_05:07:03-rgw-pacific-distro-basic-smithi/5882770/teuthology.log matches Not-understood crash indication Found 5 distinct failure reasons Failure: "2021-02-15T14:43:54.877958+0000 mon.a (mon.0) 195 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2 jobs: ['5882751', '5882733'] suites intersection: ['frontend/civetweb', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec'] suites union: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_multipart_upload}', 'tasks/rgw_user_quota}'] Failure: Command failed on smithi172 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds'" 1 jobs: ['5882610'] suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] Failure: Cannot create secret 1 jobs: ['5882598'] suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}}', 'rgw/crypt/{0-cluster/fixed-1'] Crash: "2021-02-15T16:48:57.799345+0000 mon.a (mon.0) 846 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log ceph version 16.1.0-287-g1c2f7607 (1c2f7607ce50b13c895483f8ce2ed014b62eec26) pacific (rc) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7fd152720dc0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x8ff8b) [0x7fd151db0f8b] 5: /lib64/libstdc++.so.6(+0x963ec) [0x7fd151db73ec] 6: /lib64/libstdc++.so.6(+0x96447) [0x7fd151db7447] 7: /lib64/libstdc++.so.6(+0x966a8) [0x7fd151db76a8] 8: ceph-osd(+0x587fa6) [0x55ccfd6a5fa6] 9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned int)+0x26) [0x55ccfdfb9d76] 10: (ceph::buffer::v15_2_0::create(unsigned int)+0x26) [0x55ccfdfb9dc6] 11: (ceph::buffer::v15_2_0::list::obtain_contiguous_space(unsigned int)+0xab) [0x55ccfdfb9fcb] 12: (ECSubReadReply::encode(ceph::buffer::v15_2_0::list&) const+0xfd) [0x55ccfe05592d] 13: (MOSDECSubOpReadReply::encode_payload(unsigned long)+0x65) [0x55ccfdb57465] 14: (Message::encode(unsigned long, int, bool)+0x2e) [0x55ccfdf241de] 15: (ProtocolV2::prepare_send_message(unsigned long, Message*)+0x44) [0x55ccfe176654] 16: (ProtocolV2::send_message(Message*)+0x3ae) [0x55ccfe176cbe] 17: (AsyncConnection::send_message(Message*)+0x49e) [0x55ccfe154b0e] 18: (ECBackend::_handle_message(boost::intrusive_ptr)+0x4d9) [0x55ccfdb55b89] 19: (PGBackend::handle_message(boost::intrusive_ptr)+0x52) [0x55ccfd95e312] 20: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5de) [0x55ccfd90160e] 21: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x309) [0x55ccfd78bd79] 22: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x68) [0x55ccfd9e8038] 23: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x55ccfd7ab1c8] 24: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55ccfde10514] 25: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55ccfde131b4] 26: /lib64/libpthread.so.0(+0x82de) [0x7fd1527162de] 27: clone() 1 jobs: ['5882695'] suites: ['cls', 'clusters/fixed-2', 'frontend/civetweb', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'tasks/{0-install', 'validater/valgrind}'] Failure: Command failed on smithi041 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' 1 jobs: ['5882770'] suites: ['cls', 'clusters/fixed-2', 'frontend/beast', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/valgrind}']