2022-03-09T06:51:37.087 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-09T06:51:37.091 INFO:teuthology.results:subset: '1/3'
2022-03-09T06:51:37.091 INFO:teuthology.results:seed: '3115'
2022-03-09T07:24:34.709 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-09T07:24:34.743 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-09T14:04:06.007 INFO:teuthology.results:Tests finished! gathering results...
2022-03-09T14:04:09.700 INFO:scrape:Found 50 jobs
2022-03-09T14:04:10.346 INFO:scrape:Found 4 distinct failure reasons
2022-03-09T14:04:10.347 INFO:scrape:Dead: b'2022-03-09T13:53:10.113 DEBUG:teuthology.exit:Finished running handlers'
2022-03-09T14:04:10.347 INFO:scrape:14 jobs: ['6728090', '6728076', '6728089', '6728063', '6728085', '6728092', '6728097', '6728086', '6728074', '6728091', '6728072', '6728048', '6728078', '6728057']
2022-03-09T14:04:10.348 INFO:scrape:suites intersection: []
2022-03-09T14:04:10.348 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '1-rook', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '2-workload/none', '3-final', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/3-node', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'k8s/1.21', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'net/host', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-partial-recovery}', 'rook/1.7.2}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-snaps-balanced}', 'workloads/cache}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/redirect_promote_tests}', 'workloads/redirect}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}']
2022-03-09T14:04:10.349 INFO:scrape:
2022-03-09T14:04:10.349 INFO:scrape:Crash: reached maximum tries (500) after waiting for 3000 seconds
2022-03-09T14:04:10.350 INFO:scrape:ceph version 17.0.0-10966-g5d0869c5 (5d0869c5cfb0669d6f72dda98bcf347c704046a7) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9b6bb29b20]
 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set<eversion_t, std::less<eversion_t>, std::allocator<eversion_t> >&&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, pg_missing_set<true> const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0x669) [0x55e031a31ca9]
 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set<true> const&, bool, bool*)+0x130) [0x55e031a33b80]
 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, pg_missing_set<false>&)+0x493) [0x55e0318dd583]
 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1639) [0x55e0318e5e29]
 6: main()
 7: __libc_start_main()
 8: _start()
2022-03-09T14:04:10.350 INFO:scrape:1 jobs: ['6728084']
2022-03-09T14:04:10.351 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}']
2022-03-09T14:04:10.351 INFO:scrape:
2022-03-09T14:04:10.352 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds
2022-03-09T14:04:10.353 INFO:scrape:1 jobs: ['6728083']
2022-03-09T14:04:10.353 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/radosbench-high-concurrency}']
2022-03-09T14:04:10.354 INFO:scrape:
2022-03-09T14:04:10.354 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-03-09T14:04:10.355 INFO:scrape:1 jobs: ['6728082']
2022-03-09T14:04:10.355 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}']
2022-03-09T14:04:10.355 INFO:scrape: