2022-03-01T17:46:22.834 INFO:root:teuthology version: 1.1.0-cb5b1a75
2022-03-01T17:46:22.835 INFO:teuthology.results:subset: '111/120000'
2022-03-01T17:46:22.835 INFO:teuthology.results:seed: '1332'
2022-03-01T19:01:45.109 INFO:root:teuthology version: 1.1.0-cb5b1a75
2022-03-01T19:01:45.153 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-02T01:36:16.241 INFO:teuthology.results:Tests finished! gathering results...
2022-03-02T01:36:23.532 INFO:scrape:Found 78 jobs
2022-03-02T01:36:26.613 INFO:scrape:Found 8 distinct failure reasons
2022-03-02T01:36:26.614 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2022-03-02T01:36:26.615 INFO:scrape:56 jobs
2022-03-02T01:36:26.616 INFO:scrape:suites intersection: []
2022-03-02T01:36:26.616 INFO:scrape:suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/off', 'd-balancer/upmap', 'd-thrash/default/{default', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rados/verify/{centos_latest', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_18.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_cls_all', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'ubuntu_18.04}', 'validater/lockdep}', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-radosbench}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunk-promote-flush}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}']
2022-03-02T01:36:26.617 INFO:scrape:
2022-03-02T01:36:26.618 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-03-02T01:36:26.618 INFO:scrape:1 jobs: ['6714591']
2022-03-02T01:36:26.619 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/off', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/rados_api_tests}']
2022-03-02T01:36:26.621 INFO:scrape:
2022-03-02T01:36:26.621 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2022-03-02T01:36:26.622 INFO:scrape:2 jobs: ['6714630', '6714596']
2022-03-02T01:36:26.622 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}']
2022-03-02T01:36:26.623 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}']
2022-03-02T01:36:26.623 INFO:scrape:
2022-03-02T01:36:26.624 INFO:scrape:Failure: SSH connection to smithi150 was lost: 'uname -r'
2022-03-02T01:36:26.624 INFO:scrape:3 jobs: ['6714623', '6714625', '6714626']
2022-03-02T01:36:26.624 INFO:scrape:suites intersection: ['rados']
2022-03-02T01:36:26.625 INFO:scrape:suites union: ['ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'fast/fast', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_18.04}', 'supported-random-distro$/{ubuntu_18.04}}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']
2022-03-02T01:36:26.625 INFO:scrape:
2022-03-02T01:36:26.626 INFO:scrape:Crash: timed out waiting for mon to be updated with osd.0: 60129542195 < 60129542196
2022-03-02T01:36:26.626 INFO:scrape:ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)
 1: (()+0x153c0) [0x7f104d2f63c0]
 2: (gsignal()+0xcb) [0x7f104cde118b]
 3: (abort()+0x12b) [0x7f104cdc0859]
 4: (()+0x9e911) [0x7f104d197911]
 5: (()+0xaa38c) [0x7f104d1a338c]
 6: (()+0xaa3f7) [0x7f104d1a33f7]
 7: (()+0xaa6a9) [0x7f104d1a36a9]
 8: (()+0xe1493) [0x55b59a54a493]
 9: (unsigned long const ceph::common::ConfigProxy::get_val<unsigned long>(std::basic_string_view<char, std::char_traits<char> >) const+0x8d) [0x55b59a5d502d]
 10: (()+0x1cf8d4) [0x55b59a6388d4]
 11: (DaemonServer::adjust_pgs()+0x34c) [0x55b59a63a36c]
 12: (DaemonServer::tick()+0x47) [0x55b59a651317]
 13: (Context::complete(int)+0xd) [0x55b59a5ecbdd]
 14: (SafeTimer::timer_thread()+0x1ab) [0x7f104ddbb7eb]
 15: (SafeTimerThread::entry()+0x11) [0x7f104ddbcc91]
 16: (()+0x9609) [0x7f104d2ea609]
 17: (clone()+0x43) [0x7f104cebd293]
2022-03-02T01:36:26.627 INFO:scrape:3 jobs: ['6714560', '6714569', '6714631']
2022-03-02T01:36:26.628 INFO:scrape:suites intersection: ['rados']
2022-03-02T01:36:26.628 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/fastclose', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/default', 'thrashers/mapgap', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2022-03-02T01:36:26.629 INFO:scrape:
2022-03-02T01:36:26.629 INFO:scrape:Crash: Command failed (workunit test mon/pg_autoscaler.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=45ece157fc29b55a5ee952c81c0c23abdb89bf98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pg_autoscaler.sh'
2022-03-02T01:36:26.629 INFO:scrape:ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)
 1: (()+0x12b20) [0x7fe3aeda5b20]
 2: (gsignal()+0x10f) [0x7fe3ad7f67ff]
 3: (abort()+0x127) [0x7fe3ad7e0c35]
 4: (()+0x9009b) [0x7fe3ae1ac09b]
 5: (()+0x9653c) [0x7fe3ae1b253c]
 6: (()+0x96597) [0x7fe3ae1b2597]
 7: (()+0x967f8) [0x7fe3ae1b27f8]
 8: (()+0x10d1d0) [0x55ff9bcf21d0]
 9: (unsigned long const md_config_t::get_val<unsigned long>(ConfigValues const&, std::basic_string_view<char, std::char_traits<char> >) const+0x91) [0x55ff9bd1cc01]
 10: (DaemonServer::adjust_pgs()+0x2af6) [0x55ff9bd9f636]
 11: (DaemonServer::tick()+0x103) [0x55ff9bdb2b13]
 12: (Context::complete(int)+0xd) [0x55ff9bd56ffd]
 13: (SafeTimer::timer_thread()+0x1b7) [0x7fe3b09de1c7]
 14: (SafeTimerThread::entry()+0x11) [0x7fe3b09df7a1]
 15: (()+0x814a) [0x7fe3aed9b14a]
 16: (clone()+0x43) [0x7fe3ad8bbf23]
2022-03-02T01:36:26.630 INFO:scrape:4 jobs: ['6714616', '6714602', '6714571', '6714587']
2022-03-02T01:36:26.631 INFO:scrape:suites intersection: ['rados']
2022-03-02T01:36:26.631 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton/{all/pg-autoscaler-progress-off', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}']
2022-03-02T01:36:26.632 INFO:scrape:
2022-03-02T01:36:26.632 INFO:scrape:Dead: b'2022-03-02T00:28:26.572 DEBUG:teuthology.exit:Finished running handlers'
2022-03-02T01:36:26.633 INFO:scrape:5 jobs: ['6714562', '6714620', '6714580', '6714629', '6714572']
2022-03-02T01:36:26.633 INFO:scrape:suites intersection: []
2022-03-02T01:36:26.633 INFO:scrape:suites union: ['1-install/luminous-v1only', '1-install/mimic-v1only', '1-install/nautilus-v1only', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/off', 'distro$/{ubuntu_18.04}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/standalone/{supported-random-distro$/{centos_8}', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/osd}', 'workloads/radosbench}', 'workloads/snaps-few-objects}']
2022-03-02T01:36:26.634 INFO:scrape:
2022-03-02T01:36:26.634 INFO:scrape:Crash: Command failed (workunit test mon/pg_autoscaler.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=45ece157fc29b55a5ee952c81c0c23abdb89bf98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pg_autoscaler.sh'
2022-03-02T01:36:26.635 INFO:scrape:ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)
 1: (()+0x12980) [0x7f8c134d6980]
 2: (gsignal()+0xc7) [0x7f8c125cefb7]
 3: (abort()+0x141) [0x7f8c125d0921]
 4: (()+0x8c957) [0x7f8c12fc3957]
 5: (()+0x92ae6) [0x7f8c12fc9ae6]
 6: (()+0x92b21) [0x7f8c12fc9b21]
 7: (()+0x92d54) [0x7f8c12fc9d54]
 8: (()+0xe9ff0) [0x55656687dff0]
 9: (unsigned long const ceph::common::ConfigProxy::get_val<unsigned long>(std::basic_string_view<char, std::char_traits<char> >) const+0xd9) [0x5565668cf5f9]
 10: (DaemonServer::adjust_pgs()+0x1682) [0x55656692bde2]
 11: (DaemonServer::tick()+0x45) [0x55656693f255]
 12: (Context::complete(int)+0x9) [0x5565668e4919]
 13: (SafeTimer::timer_thread()+0x1ac) [0x7f8c1430a08c]
 14: (SafeTimerThread::entry()+0xd) [0x7f8c1430b45d]
 15: (()+0x76db) [0x7f8c134cb6db]
 16: (clone()+0x3f) [0x7f8c126b171f]
2022-03-02T01:36:26.635 INFO:scrape:1 jobs: ['6714618']
2022-03-02T01:36:26.636 INFO:scrape:suites: ['msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/pg-autoscaler', 'supported-random-distro$/{ubuntu_18.04}}']
2022-03-02T01:36:26.636 INFO:scrape: