2021-08-11T18:58:34.057 INFO:root:teuthology version: 1.1.0-321319b1
2021-08-11T18:58:34.059 INFO:teuthology.results:subset: '2/3'
2021-08-11T18:58:34.059 INFO:teuthology.results:seed: '2822'
2021-08-11T19:58:16.995 INFO:root:teuthology version: 1.1.0-321319b1
2021-08-11T19:58:17.047 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-08-12T07:59:09.539 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-08-12T07:59:12.597 INFO:scrape:Found 24 jobs
2021-08-12T07:59:13.578 INFO:scrape:Found 7 distinct failure reasons
2021-08-12T07:59:13.579 INFO:scrape:Dead: b'2021-08-12T07:59:12.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight'
2021-08-12T07:59:13.579 INFO:scrape:6 jobs: ['6335149', '6335138', '6335142', '6335144', '6335140', '6335135']
2021-08-12T07:59:13.580 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'msgr-failures/osd-dispatch-delay', 'openstack}', 'rados', 'thrashers/mapgap', 'thrashosds-health']
2021-08-12T07:59:13.580 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps}', 'workloads/dedup-io-snaps}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}']
2021-08-12T07:59:13.581 INFO:scrape:
2021-08-12T07:59:13.581 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2021-08-12T07:59:13.582 INFO:scrape:1 jobs: ['6335146']
2021-08-12T07:59:13.582 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/rados_api_tests}']
2021-08-12T07:59:13.583 INFO:scrape:
2021-08-12T07:59:13.583 INFO:scrape:Dead: b'2021-08-12T07:59:12.611 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops'
2021-08-12T07:59:13.583 INFO:scrape:1 jobs: ['6335137']
2021-08-12T07:59:13.584 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}']
2021-08-12T07:59:13.584 INFO:scrape:
2021-08-12T07:59:13.585 INFO:scrape:Dead: b'2021-08-12T07:14:32.443 DEBUG:teuthology.exit:Finished running handlers'
2021-08-12T07:59:13.585 INFO:scrape:5 jobs: ['6335129', '6335130', '6335131', '6335127', '6335128']
2021-08-12T07:59:13.586 INFO:scrape:suites intersection: ['1-pg-log-overrides/short_pg_log', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health']
2021-08-12T07:59:13.587 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/small-objects-balanced}']
2021-08-12T07:59:13.587 INFO:scrape:
2021-08-12T07:59:13.588 INFO:scrape:Dead: b'OSError: Socket is closed'
2021-08-12T07:59:13.588 INFO:scrape:7 jobs: ['6335134', '6335143', '6335141', '6335147', '6335136', '6335150', '6335148']
2021-08-12T07:59:13.589 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'msgr-failures/osd-dispatch-delay', 'openstack}', 'rados', 'thrashers/mapgap', 'thrashosds-health']
2021-08-12T07:59:13.589 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/dedup-io-mixed}', 'workloads/pool-snaps-few-objects}', 'workloads/redirect_set_object}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects}']
2021-08-12T07:59:13.590 INFO:scrape:
2021-08-12T07:59:13.590 INFO:scrape:Failure: reached maximum tries (800) after waiting for 4800 seconds
2021-08-12T07:59:13.591 INFO:scrape:3 jobs: ['6335139', '6335133', '6335132']
2021-08-12T07:59:13.591 INFO:scrape:suites intersection: ['1-pg-log-overrides/short_pg_log', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health']
2021-08-12T07:59:13.592 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench}']
2021-08-12T07:59:13.592 INFO:scrape:
2021-08-12T07:59:13.593 INFO:scrape:Crash: reached maximum tries (500) after waiting for 3000 seconds
2021-08-12T07:59:13.593 INFO:scrape:ceph version 17.0.0-6620-gc53253a7 (c53253a765ab5f3b037450ebf4e246a9d9702d12) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9dfc8413c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x2bb) [0x56313fb1e6aa]
 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x56313fb1e772]
 6: (PgScrubber::scrub_finish()+0x1ed5) [0x56313f70df91]
 7: (PgScrubber::on_digest_updates()+0x7e7) [0x56313f71145f]
 8: (Scrub::WaitDigestUpdate::react(Scrub::DigestUpdate const&)+0xae) [0x56313f720802]
 9: (boost::statechart::simple_state<Scrub::WaitDigestUpdate, Scrub::ActiveScrubbing, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xe4) [0x56313f72829a]
 10: (boost::statechart::state_machine<Scrub::ScrubMachine, Scrub::NotActive, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x3c1) [0x56313f716035]
 11: (PgScrubber::send_maps_compared(unsigned int)+0x8f) [0x56313f6f63e9]
 12: (PG::forward_scrub_event(void (ScrubPgIF::*)(unsigned int), unsigned int, std::basic_string_view<char, std::char_traits<char> >)+0x6d) [0x56313f377aad]
 13: (ceph::osd::scheduler::PGScrubMapsCompared::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x32) [0x56313f5a8784]
 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x42f8) [0x56313f2c449c]
 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x96a) [0x56313fb0cf7a]
 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x17) [0x56313fb0f553]
 17: (Thread::entry_wrapper()+0x43) [0x56313faffc33]
 18: (Thread::_entry_func(void*)+0xd) [0x56313faffc4f]
 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f9dfc835609]
 20: clone()
2021-08-12T07:59:13.594 INFO:scrape:1 jobs: ['6335145']
2021-08-12T07:59:13.594 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench}']
2021-08-12T07:59:13.595 INFO:scrape: