2023-09-03T10:46:14.144 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556
2023-09-03T10:46:14.145 INFO:teuthology.results:subset: '1/1000'
2023-09-03T10:46:14.145 INFO:teuthology.results:seed: '9432'
2023-09-03T11:03:26.239 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556
2023-09-03T11:03:26.242 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-03T23:04:19.010 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-09-03T23:04:20.772 INFO:scrape:Found 25 jobs
2023-09-03T23:04:21.182 INFO:scrape:Found 11 distinct failure reasons
2023-09-03T23:04:21.183 INFO:scrape:Dead: b'2023-09-03T23:04:20.730 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops'
2023-09-03T23:04:21.184 INFO:scrape:1 jobs: ['7387793']
2023-09-03T23:04:21.184 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}']
2023-09-03T23:04:21.184 INFO:scrape:
2023-09-03T23:04:21.185 INFO:scrape:Dead: b'2023-09-03T23:04:20.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops'
2023-09-03T23:04:21.185 INFO:scrape:1 jobs: ['7387782']
2023-09-03T23:04:21.186 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/small-objects-localized}']
2023-09-03T23:04:21.186 INFO:scrape:
2023-09-03T23:04:21.187 INFO:scrape:Failure: Host key for server 'smithi064.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHTpXP23kOBdWl0stsA1+BamA76xMLM3yoS6aDSEbLJ1', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC9t8dmJD3H2RoGKvUak1hZcYvhddPUxzuuRbRhtC6E4IJD0P+D3ZlPZq5byq3S0iQofKEXdSWUZnbJ3PR5XxBYdHbLk/W2Z4rPWfdD9IgUQ/lH4LTm+BjnimQazaUK74FD3hgzbF+CjnGZU+SCilKp5DaBcWJEC03R5p1A6EPfoVCK9qub7/F14K4k2tXwo6cRkUdo/zp2Jxck8xZx9fP0ZWLJ7D3IoBQGr3/kYCeI7Q6F66GDw63W4VG8K0U5+pFPJcOdG0OtECW3x3EBNHu30zGTnB87T2ub9ViOMxyTXv4GXk3TtazN7yAaeG5TzkeJv5izHZyAvi3zJ3DUKIYyFf9dkeVacZQo51cTzZdhXoqWAjzsX2UGEGCghmt0GkeLcah+lRw0w9HzZM83IXOG/Uru6++P5upaWPZVagNbRNiJg5S1MXQlkQo+6ac0gjqH+DI3SGYKmB2yM/xwvGLHAgr/baJkp/8gwcC4JdYIzuNaTcMI7HQz/BXivthEr4s='
2023-09-03T23:04:21.187 INFO:scrape:1 jobs: ['7387787']
2023-09-03T23:04:21.187 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/dedup-io-snaps}']
2023-09-03T23:04:21.188 INFO:scrape:
2023-09-03T23:04:21.188 INFO:scrape:Dead: b'2023-09-03T23:04:10.365 DEBUG:teuthology.exit:Finished running handlers'
2023-09-03T23:04:21.189 INFO:scrape:4 jobs: ['7387780', '7387775', '7387777', '7387778']
2023-09-03T23:04:21.189 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health']
2023-09-03T23:04:21.190 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/small-objects}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}']
2023-09-03T23:04:21.190 INFO:scrape:
2023-09-03T23:04:21.190 INFO:scrape:Failure: Host key for server 'smithi175.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIeeBlFwklOx2PAuzbOpnOrhhkB6sAsvu30qxdaU1Nud', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDaMvrPC20DkESkomhF0eSj9rynyId3l2Od+KRovVeVro0IddQIl/ZIpH/pMBdkDl/ytypbs5DJj8kbmWTVMMM9e8V9U32p/V5JLc9LmBt5BDWdlCQgMox5ep3TDQ8LsitQACOlgaMPbVX+EX+iEac7QEoK+1LPV3hg5oJI7mDBQ9kuqLs4HdVEl3yVs5+wE6bKoqrZwhGHTAvCuPEFcWbJi3jlfWE5N1Haoq8D0uPyb8euT3K8Lzp5tZd8kyyWASONpZ72OeKP/NhRaV/m0SW6WKZ+ItLmX7y/F/QVub67QYQRWFTOREATX0LUVYJ0kGLrIrBUMxnQj6O6Hmx+v8tSLCVmlhePllGd13RWDLwbkcA+uToQxS+8P0lWdS8Y4uQcl18Qp4mKW6066gNjkspi/ZyVKNezdAVqFzZa/nYKgvhhIGZdRCjuK1joX0p8ZsKp9S5LBkw12SYb405HOrd6pFkP+2dLZn7H4RAb3j098hUFLuEBuDxMiBITl818UjU='
2023-09-03T23:04:21.191 INFO:scrape:1 jobs: ['7387792']
2023-09-03T23:04:21.191 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}']
2023-09-03T23:04:21.192 INFO:scrape:
2023-09-03T23:04:21.192 INFO:scrape:Dead: b'2023-09-03T23:04:20.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops'
2023-09-03T23:04:21.192 INFO:scrape:1 jobs: ['7387788']
2023-09-03T23:04:21.193 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']
2023-09-03T23:04:21.193 INFO:scrape:
2023-09-03T23:04:21.194 INFO:scrape:Dead: b'OSError: Socket is closed'
2023-09-03T23:04:21.194 INFO:scrape:2 jobs: ['7387791', '7387785']
2023-09-03T23:04:21.194 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-2}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'supported-random-distro$/{centos_latest}', 'thrashosds-health']
2023-09-03T23:04:21.195 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/small-objects}', 'workloads/snaps-few-objects-localized}']
2023-09-03T23:04:21.195 INFO:scrape:
2023-09-03T23:04:21.196 INFO:scrape:Failure: Host key for server 'smithi019.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIF1QHPnxaTDV8rXSpRF9A86lW9AWt5+h1+GCfjKppRww', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDGN/6VmweNYha/ewqF052lUltW++J7AL4brKQlRAKPNDnH8JiO3P2OTwVau2eDQknv7uU9saelphzStC8TDHEx2Toz/Y4f3ors+ortXOjUEqo9MJBo2T5JhqO/o+JQZOohPcHoqopMLK/PqWE9cTgrNiQPq0uNlqfqSkOvxOwVVDWYnno42APafc1z58zRlAbPRm9m6Fi+tWzdtqQBK8MpzAOtv78TQ5esB5BFWhZ5JOHD1QK25WUfItZQkAtgGA+Afr4LJJzv+nPVSTGYSyeDQoIZhfADAjIZo4amjeL4Ah2gzNQ/IwC7gPruRcXhsbaF19JCe4y45WO7yqkI2azRoW5UFs82hSoMlvC4dFvo05wZxFBKeoGzI1zpXvcIx+1RvvZ+lw8HaLJ54d4avYlnN3zMRMqXC7BZnPhnI4f3XAKB66iZV0BMtp4SpnhrfZAzy+S35LXKyTfmDXojZHf7CNV8uirNA5RzwaJBsWhRZuqKPhEWFX+GDcv7O7HZ/nc='
2023-09-03T23:04:21.196 INFO:scrape:1 jobs: ['7387771']
2023-09-03T23:04:21.196 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/dedup-io-snaps}']
2023-09-03T23:04:21.197 INFO:scrape:
2023-09-03T23:04:21.197 INFO:scrape:Failure: Host key for server 'smithi072.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIuk722LE1W6TS1fXbeZycJneK/jSEpezpLb1vzyYB7Q', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDL788vJUQYDgz7cz7j/qSnmZ3PfLKVaLsVyNAz9qyi6ycSjEvmeONey6+VqluyjVDo6JoF1BJjsLiyLd/ulQFG25j2Ll9ges4yJbCOlwiutcbDn1QLSVtblBt8w7CMb/LaSJu89vf4dtapBEIbpdOzb1AU2PwPsjUpjyP85Imp9SzhSzP//UW+NVc5f36dJlLXNlaVoZpRTUq4zfjgHAnTBIaj4j8CmKmqv8HBPN0i46Pf6dUukLjm79bI4KkM2UgxBeSWTbUZ9nuu2iylVoIOXYaEMYI/YXyp+CS8qFPT0+ghOd8QNMjXVZlf/k6fK9Dm742lLRf68krq88uCst8baU8ymfldjLvSkMrmfPXoRswPXd38j4FzXoKE7dVWxC5Xoz5xkKvQPKtTb33uT9MkSjEji/96jdxmRa8JkDi29B9J0qpyx795yKslFBLWJ9O4obwQlBg1ZHrUn129sPhSaL4mK/98u9Uu7J+WqW+pYh1Z02rmSvShQGLkZXgi0CM='
2023-09-03T23:04:21.198 INFO:scrape:1 jobs: ['7387772']
2023-09-03T23:04:21.198 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']
2023-09-03T23:04:21.199 INFO:scrape:
2023-09-03T23:04:21.199 INFO:scrape:Failure: Host key for server 'smithi155.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIP58k6WKOBqj07h8qyYnUGsFd5XdSF0R1c7lsfW4abYF', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDs+7yprc3/VKyOjbI1Jg0q2NuvDO1XMr3guUseNyqTZ97k9LQlO2AXz2SM8BKQ73+kLv/Hi8GedxG1P8wYt3PavLVCMqxwNHvE4K3BG8S1cyAtZFV7zvScYpxwjbVAXyo4Tt0RMmB3snagsCLHkAHZFk3gTRanqTiBOM6DIJnN7RT66Lq1ja0iqAwP6o2X+hqzAg6NXYbp3ENVuGVl1D4tCpkceulW/Ab1EESZhkel9yWAcJiCewR/un5mC2eD4iHPYnMyvLwO2vHE5UBTQ3mbK622rklCsYiVd7RWc6Mh3h71SeHUES8OfOUs9JudWBDiYAL2vEDosb6H0LKfrGT/BiSDOK/pxGGpn2vlngURcCNk1lxiB/Eq04gR7OpdE5JcvaFvgofJTd5KZbuZX4s2Kn7F+64uAOqZDcCyPo2CAodz6K0DdeP8Y9GixwSbeagF4C10VunwJaQNkXsuWXX1S4H68QC3pBTFIDeXJ6lOjGxiU65N1uD6Ay3NtlXRW3c='
2023-09-03T23:04:21.200 INFO:scrape:1 jobs: ['7387786']
2023-09-03T23:04:21.200 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects}']
2023-09-03T23:04:21.200 INFO:scrape:
2023-09-03T23:04:21.201 INFO:scrape:Failure: Host key for server 'smithi096.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINSFIxIkJsnT6mPAFfGKFL6FpM9K3jlBWrJLNDzarvCH', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCqMCKpRjytxUCj6jlFN7GSVXX4LYBBYwiLlS8cHnc+uYVH56JiOt8mlzR83lFLMQs49/3q6ZQOBahZ0wJrbtlm2cX2Z1VJ58/47WYy5LmTDyPJujBt6N4LaOCoHRp9R7ddvG6Ku1ZE2rnZ86K9huIMH29blNmThNpa+pbYHl3F60dLCr3YgkwYvpO9JRVKUgAOiyyn2eCmpQ1rcs++dp/QT2sctdmMkgYKrpEmSWF5bmnDBsGpk80/hO/52MPqrSjKiskGlaqF/8zzAgcngxo/vY78QLuoj9TmtSe9K1aIlGgniUiM+1PzIFN1p/wDuFaw3MfYkFWWbkPNB8JyjOFc5rHwQOotm7QGcFLSpQWcCB6on+/hz/8J/vBFtSeolosYKcXyTSMPXxwHMA3w0vR6QtZWgL8oV6Espv78hkXVuB8xHD8suKa+mH8CpSlybRUxFrSVtWzAxm1mtnee5SVC/HgdhdcUOa60+2OvMG+YlritcJRCGUPap1e73z3lGLc='
2023-09-03T23:04:21.201 INFO:scrape:1 jobs: ['7387795']
2023-09-03T23:04:21.201 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/dedup-io-snaps}']
2023-09-03T23:04:21.202 INFO:scrape: