2021-02-10T04:03:21.101 INFO:root:teuthology version: 1.1.0-14fad317
2021-02-10T04:03:21.102 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-02-10T16:03:46.796 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-02-10T16:14:00.785 INFO:scrape:Found 356 jobs
2021-02-10T16:15:53.287 INFO:scrape:Found 12 distinct failure reasons
2021-02-10T16:15:53.477 INFO:scrape:Dead: b'2021-02-10T15:27:55.770 DEBUG:teuthology.exit:Finished running handlers'
2021-02-10T16:15:53.477 INFO:scrape:13 jobs: ['5871729', '5871551', '5871442', '5871719', '5871463', '5871632', '5871777', '5871639', '5871621', '5871590', '5871473', '5871755', '5871758']
2021-02-10T16:15:53.477 INFO:scrape:suites intersection: []
2021-02-10T16:15:53.478 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'scheduler/dmclock_1Shard_16Threads', 'scheduler/dmclock_default_shards', 'settings/optimized', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'ubuntu_latest', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-small-objects}', 'workloads/redirect_promote_tests}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}']
2021-02-10T16:15:53.478 INFO:scrape:
2021-02-10T16:15:53.478 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed.
2021-02-10T16:15:53.478 INFO:scrape:8 jobs: ['5871605', '5871691', '5871750', '5871595', '5871613', '5871762', '5871781', '5871573']
2021-02-10T16:15:53.478 INFO:scrape:suites intersection: ['rados']
2021-02-10T16:15:53.479 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/osd-recovery', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}']
2021-02-10T16:15:53.479 INFO:scrape:
2021-02-10T16:15:53.479 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d71c4615880b1597d35f490412cfa10133b71bdf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2021-02-10T16:15:53.479 INFO:scrape:1 jobs: ['5871457']
2021-02-10T16:15:53.479 INFO:scrape:suites: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
2021-02-10T16:15:53.479 INFO:scrape:
2021-02-10T16:15:53.479 INFO:scrape:Failure: Command failed on smithi090 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71c4615880b1597d35f490412cfa10133b71bdf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 15efc7e2-6b4d-11eb-8fea-001a4aab830c -- ceph osd crush tunables default'
2021-02-10T16:15:53.479 INFO:scrape:1 jobs: ['5871715']
2021-02-10T16:15:53.480 INFO:scrape:suites: ['rados/cephadm/smoke-roleless/{distro/rhel_latest', 'start}']
2021-02-10T16:15:53.480 INFO:scrape:
2021-02-10T16:15:53.480 INFO:scrape:Failure: Command failed on smithi131 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7a0f8f0-6b29-11eb-8fe8-001a4aab830c -e sha1=d71c4615880b1597d35f490412cfa10133b71bdf -- bash -c 'ceph orch ps'"
2021-02-10T16:15:53.480 INFO:scrape:2 jobs: ['5871454', '5871631']
2021-02-10T16:15:53.480 INFO:scrape:suites intersection: ['1-tasks', 'distro1$/{centos_8.0}', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2021-02-10T16:15:53.480 INFO:scrape:suites union: ['1-tasks', 'distro1$/{centos_8.0}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2021-02-10T16:15:53.480 INFO:scrape:
2021-02-10T16:15:53.480 INFO:scrape:Failure: Command failed on smithi089 with status 1: 'sudo docker kill -s 1 ceph-d99e5f66-6b2d-11eb-8fe8-001a4aab830c-osd.7'
2021-02-10T16:15:53.481 INFO:scrape:5 jobs: ['5871477', '5871561', '5871597', '5871658', '5871623']
2021-02-10T16:15:53.481 INFO:scrape:suites intersection: ['ceph', 'clusters/{openstack', 'distro$/{ubuntu_18.04}', 'rados', 'thrashosds-health', 'three-plus-one}']
2021-02-10T16:15:53.482 INFO:scrape:suites union: ['1-install/luminous', '1-install/mimic', '1-install/nautilus-v1only', '1-install/pacific', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/snaps-few-objects}']
2021-02-10T16:15:53.482 INFO:scrape:
2021-02-10T16:15:53.482 INFO:scrape:Failure: 
2021-02-10T16:15:53.482 INFO:scrape:1 jobs: ['5871713']
2021-02-10T16:15:53.482 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{rhel_8}', 'tasks/scrub_test}']
2021-02-10T16:15:53.482 INFO:scrape:
2021-02-10T16:15:53.482 INFO:scrape:Failure: wait_for_clean: failed before timeout expired
2021-02-10T16:15:53.483 INFO:scrape:1 jobs: ['5871505']
2021-02-10T16:15:53.483 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/lost-unfound-delete', 'supported-random-distro$/{rhel_8}}']
2021-02-10T16:15:53.483 INFO:scrape:
2021-02-10T16:15:53.483 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2021-02-10T16:15:53.483 INFO:scrape:1 jobs: ['5871480']
2021-02-10T16:15:53.483 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2021-02-10T16:15:53.483 INFO:scrape:
2021-02-10T16:15:53.483 INFO:scrape:Crash: wait_for_clean: failed before timeout expired
2021-02-10T16:15:53.484 INFO:scrape:ceph version 17.0.0-344-gd71c4615 (d71c4615880b1597d35f490412cfa10133b71bdf) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7fa18e615b20]
 2: pthread_cond_timedwait()
 3: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x355) [0x55656edb64b5]
 4: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55656f41aca4]
 5: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55656f41d944]
 6: (Thread::_entry_func(void*)+0xd) [0x55656f40c85d]
 7: /lib64/libpthread.so.0(+0x814a) [0x7fa18e60b14a]
 8: clone()
2021-02-10T16:15:53.484 INFO:scrape:1 jobs: ['5871516']
2021-02-10T16:15:53.484 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/lost-unfound', 'supported-random-distro$/{rhel_8}}']
2021-02-10T16:15:53.484 INFO:scrape:
2021-02-10T16:15:53.484 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d71c4615880b1597d35f490412cfa10133b71bdf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2021-02-10T16:15:53.484 INFO:scrape:1 jobs: ['5871787']
2021-02-10T16:15:53.485 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}']
2021-02-10T16:15:53.485 INFO:scrape:
2021-02-10T16:15:53.485 INFO:scrape:Failure: "2021-02-10T02:18:54.333419+0000 mon.a (mon.0) 6537 : cluster [WRN] Health check failed: Telemetry requires re-opt-in (TELEMETRY_CHANGED)" in cluster log
2021-02-10T16:15:53.485 INFO:scrape:1 jobs: ['5871628']
2021-02-10T16:15:53.485 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/dashboard}']
2021-02-10T16:15:53.485 INFO:scrape: