2020-08-26T05:53:40.022 INFO:root:teuthology version: 1.0.0-3b2e857 2020-08-26T05:53:40.024 INFO:teuthology.results:subset: '0/1000' 2020-08-26T05:53:40.025 INFO:teuthology.results:seed: '2212' 2020-08-26T06:24:24.346 INFO:root:teuthology version: 1.0.0-3b2e857 2020-08-26T06:24:24.386 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-26T07:01:25.538 INFO:teuthology.results:Tests finished! gathering results... 2020-08-26T07:01:39.362 INFO:scrape:Found 69 jobs 2020-08-26T07:01:52.549 INFO:scrape:Found 4 distinct failure reasons 2020-08-26T07:01:52.553 INFO:scrape:Crash: "2020-08-26T06:30:32.014158+0000 mon.b (mon.0) 141 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log 2020-08-26T07:01:52.553 INFO:scrape:ceph version 16.0.0-4450-g8f84928 (8f849282202e28a81970863287b6d80ee843a28e) pacific (dev) 1: (()+0x12dc0) [0x7f9872deddc0] 2: (gsignal()+0x10f) [0x7f9871ac88df] 3: (abort()+0x127) [0x7f9871ab2cf5] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x55b9f5629daf] 5: (()+0x534f78) [0x55b9f5629f78] 6: (PrimaryLogPG::eval_repop(PrimaryLogPG::RepGather*)+0xbff) [0x55b9f584373f] 7: (PrimaryLogPG::repop_all_committed(PrimaryLogPG::RepGather*)+0x93) [0x55b9f5843a83] 8: (Context::complete(int)+0xd) [0x55b9f576533d] 9: (ReplicatedBackend::do_repop_reply(boost::intrusive_ptr)+0x1d5) [0x55b9f5a89395] 10: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x14f) [0x55b9f5a9c39f] 11: (PGBackend::handle_message(boost::intrusive_ptr)+0x52) [0x55b9f5908af2] 12: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5de) [0x55b9f58ac4fe] 13: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x309) [0x55b9f572c279] 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x68) [0x55b9f59926d8] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12ef) [0x55b9f574b63f] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b9f5d8b854] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b9f5d8e4f4] 18: (()+0x82de) [0x7f9872de32de] 19: (clone()+0x43) [0x7f9871b8d133] 2020-08-26T07:01:52.554 INFO:scrape:30 jobs 2020-08-26T07:01:52.557 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'thrashosds-health'] 2020-08-26T07:01:52.560 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2020-08-26T07:01:52.561 INFO:scrape: 2020-08-26T07:01:52.561 INFO:scrape:Failure: 500 Server Error: Internal Server Error for url: https://3.chacra.ceph.com/repos/ceph/wip-shuzhenyi-testing-2020-08-26-1136/8f849282202e28a81970863287b6d80ee843a28e/centos/8/flavors/default/repo 2020-08-26T07:01:52.562 INFO:scrape:5 jobs: ['5376770', '5376752', '5376768', '5376789', '5376779'] 2020-08-26T07:01:52.562 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2020-08-26T07:01:52.563 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2020-08-26T07:01:52.563 INFO:scrape: 2020-08-26T07:01:52.564 INFO:scrape:Crash: Command failed on smithi176 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2020-08-26T07:01:52.564 INFO:scrape:ceph version 16.0.0-4450-g8f84928 (8f849282202e28a81970863287b6d80ee843a28e) pacific (dev) 1: (()+0x128a0) [0x7f3de98448a0] 2: (gsignal()+0xc7) [0x7f3de84f6f47] 3: (abort()+0x141) [0x7f3de84f88b1] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x55d720542928] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55d720542ab2] 6: (PrimaryLogPG::eval_repop(PrimaryLogPG::RepGather*)+0xbaf) [0x55d7206e39df] 7: (PrimaryLogPG::repop_all_committed(PrimaryLogPG::RepGather*)+0x8b) [0x55d7206e3c2b] 8: (Context::complete(int)+0x9) [0x55d720601769] 9: (ReplicatedBackend::op_commit(boost::intrusive_ptr const&)+0x104) [0x55d7209209a4] 10: (Context::complete(int)+0x9) [0x55d720601769] 11: (PrimaryLogPG::BlessedContext::finish(int)+0x91) [0x55d720756d31] 12: (Context::complete(int)+0x9) [0x55d720601769] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6ac) [0x55d7205e4cfc] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55d720c2f3fc] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55d720c326b0] 16: (()+0x76db) [0x7f3de98396db] 17: (clone()+0x3f) [0x7f3de85d9a3f] 2020-08-26T07:01:52.565 INFO:scrape:2 jobs: ['5376753', '5376750'] 2020-08-26T07:01:52.565 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr/async-v1only', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health'] 2020-08-26T07:01:52.566 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}'] 2020-08-26T07:01:52.566 INFO:scrape: 2020-08-26T07:01:52.567 INFO:scrape:Crash: Command failed on smithi059 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2020-08-26T07:01:52.567 INFO:scrape:ceph version 16.0.0-4450-g8f84928 (8f849282202e28a81970863287b6d80ee843a28e) pacific (dev) 1: (()+0x128a0) [0x7f3fdfa8a8a0] 2: (gsignal()+0xc7) [0x7f3fde73cf47] 3: (abort()+0x141) [0x7f3fde73e8b1] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x562272484928] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x562272484ab2] 6: (PrimaryLogPG::eval_repop(PrimaryLogPG::RepGather*)+0xbaf) [0x5622726259df] 7: (PrimaryLogPG::repop_all_committed(PrimaryLogPG::RepGather*)+0x8b) [0x562272625c2b] 8: (Context::complete(int)+0x9) [0x562272543769] 9: (ReplicatedBackend::do_repop_reply(boost::intrusive_ptr)+0x167) [0x562272862d17] 10: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x87) [0x5622728761b7] 11: (PGBackend::handle_message(boost::intrusive_ptr)+0x97) [0x5622726f0557] 12: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6fd) [0x56227268d71d] 13: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x56227250939b] 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x562272774e37] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x562272526f5c] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x562272b713fc] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x562272b746b0] 18: (()+0x76db) [0x7f3fdfa7f6db] 19: (clone()+0x3f) [0x7f3fde81fa3f] 2020-08-26T07:01:52.568 INFO:scrape:1 jobs: ['5376765'] 2020-08-26T07:01:52.570 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects-balanced}'] 2020-08-26T07:01:52.570 INFO:scrape: