2022-06-28T02:31:51.524 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef
2022-06-28T02:31:51.526 INFO:teuthology.results:subset: '1/6'
2022-06-28T02:31:51.526 INFO:teuthology.results:seed: '843'
2022-06-28T03:11:02.711 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef
2022-06-28T03:11:02.717 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-06-28T03:55:07.413 INFO:teuthology.results:Tests finished! gathering results...
2022-06-28T03:55:17.649 INFO:scrape:Found 48 jobs
2022-06-28T03:55:17.811 INFO:scrape:Found 3 distinct failure reasons
2022-06-28T03:55:17.813 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2022-06-28T03:55:17.820 INFO:scrape:41 jobs
2022-06-28T03:55:17.822 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health']
2022-06-28T03:55:17.823 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados::thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados::thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/dedup-io-mixed}', 'workloads/dedup-io-snaps}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/write_fadvise_dontneed}']
2022-06-28T03:55:17.825 INFO:scrape:
2022-06-28T03:55:17.839 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2022-06-28T03:55:17.840 INFO:scrape:5 jobs: ['6901128', '6901111', '6901134', '6901098', '6901118']
2022-06-28T03:55:17.841 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health']
2022-06-28T03:55:17.842 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados::thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados::thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/mapgap', 'thrashers/none', 'thrashosds-health', 'workloads/dedup-io-mixed}', 'workloads/radosbench-high-concurrency}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}']
2022-06-28T03:55:17.843 INFO:scrape:
2022-06-28T03:55:17.844 INFO:scrape:Crash: Command failed on smithi040 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0'
2022-06-28T03:55:17.845 INFO:scrape:ceph version 17.0.0-13211-g0c2f649b (0c2f649b73a2c97adaf7ceeb84e3386572066ddc) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9bad04dce0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x563e5fa479ba]
 5: ceph-osd(+0x61bb7d) [0x563e5fa47b7d]
 6: (PgScrubber::update_scrub_stats(std::chrono::time_point<ceph::coarse_real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x52d) [0x563e5fd7a1fd]
 7: (PG::with_pg_stats(std::chrono::time_point<ceph::coarse_real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::function<void (pg_stat_t const&, unsigned int)>&&)+0x56) [0x563e5fc26c26]
 8: (OSD::collect_pg_stats()+0x6c5) [0x563e5fb66615]
 9: (MgrClient::_send_pgstats()+0x40) [0x563e604ba2a0]
 10: (MgrClient::_send_stats()+0x15) [0x563e604bd335]
 11: (Context::complete(int)+0xd) [0x563e5fb9424d]
 12: (CommonSafeTimer<std::mutex>::timer_thread()+0x11a) [0x563e60287f5a]
 13: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x563e60289581]
 14: /lib64/libpthread.so.0(+0x81cf) [0x7f9bad0431cf]
 15: clone()
2022-06-28T03:55:17.846 INFO:scrape:2 jobs: ['6901136', '6901140']
2022-06-28T03:55:17.847 INFO:scrape:suites intersection: ['1-pg-log-overrides/short_pg_log', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'openstack}', 'rados', 'rados::thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashosds-health']
2022-06-28T03:55:17.852 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados::thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/redirect}']
2022-06-28T03:55:17.854 INFO:scrape: