2021-02-17T09:36:05.298 INFO:root:teuthology version: 1.1.0-071a4df2
2021-02-17T09:36:05.302 INFO:teuthology.results:subset: '1/115000'
2021-02-17T09:36:05.302 INFO:teuthology.results:seed: '2477'
2021-02-17T12:26:31.622 INFO:root:teuthology version: 1.1.0-071a4df2
2021-02-17T12:26:31.660 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-02-18T00:00:55.683 INFO:teuthology.results:Tests finished! gathering results...
2021-02-18T00:24:14.444 INFO:scrape:Found 358 jobs
2021-02-18T00:29:54.723 INFO:scrape:Found 11 distinct failure reasons
2021-02-18T00:29:55.294 INFO:scrape:Dead: b'2021-02-17T23:57:27.088 DEBUG:teuthology.exit:Finished running handlers'
2021-02-18T00:29:55.294 INFO:scrape:6 jobs: ['5890847', '5890756', '5890714', '5890656', '5890749', '5890707']
2021-02-18T00:29:55.295 INFO:scrape:suites intersection: []
2021-02-18T00:29:55.296 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fixed-2', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}']
2021-02-18T00:29:55.296 INFO:scrape:
2021-02-18T00:29:55.297 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2021-02-18T00:29:55.297 INFO:scrape:1 jobs: ['5890596']
2021-02-18T00:29:55.298 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2021-02-18T00:29:55.299 INFO:scrape:
2021-02-18T00:29:55.299 INFO:scrape:Failure: Command failed on smithi185 with status 1: 'sudo docker kill -s 1 ceph-aeb23922-7108-11eb-9009-001a4aab830c-osd.11'
2021-02-18T00:29:55.300 INFO:scrape:2 jobs: ['5890592', '5890811']
2021-02-18T00:29:55.300 INFO:scrape:suites intersection: []
2021-02-18T00:29:55.301 INFO:scrape:suites union: ['1-install/mimic-v1only', '1-start', '2-thrash', '3-tasks/snaps-few-objects', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'rados', 'rados/cephadm/thrash/{0-distro/centos_latest', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'root}', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}']
2021-02-18T00:29:55.301 INFO:scrape:
2021-02-18T00:29:55.302 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735bf90a2cfdf28b8ab4913090a3aedebc37463c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2021-02-18T00:29:55.302 INFO:scrape:2 jobs: ['5890572', '5890750']
2021-02-18T00:29:55.303 INFO:scrape:suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
2021-02-18T00:29:55.303 INFO:scrape:suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
2021-02-18T00:29:55.303 INFO:scrape:
2021-02-18T00:29:55.304 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed.
2021-02-18T00:29:55.304 INFO:scrape:4 jobs: ['5890880', '5890722', '5890809', '5890712']
2021-02-18T00:29:55.305 INFO:scrape:suites intersection: ['rados']
2021-02-18T00:29:55.305 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}']
2021-02-18T00:29:55.305 INFO:scrape:
2021-02-18T00:29:55.306 INFO:scrape:Failure: wait_for_clean: failed before timeout expired
2021-02-18T00:29:55.306 INFO:scrape:1 jobs: ['5890632']
2021-02-18T00:29:55.307 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/lost-unfound', 'supported-random-distro$/{rhel_8}}']
2021-02-18T00:29:55.307 INFO:scrape:
2021-02-18T00:29:55.308 INFO:scrape:Failure: Command failed on smithi140 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1d83bcfa-7112-11eb-9009-001a4aab830c -e sha1=735bf90a2cfdf28b8ab4913090a3aedebc37463c -- bash -c 'ceph orch ps'"
2021-02-18T00:29:55.308 INFO:scrape:2 jobs: ['5890748', '5890569']
2021-02-18T00:29:55.308 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2021-02-18T00:29:55.309 INFO:scrape:suites union: ['1-tasks', 'distro1$/{centos_latest}', 'distro1$/{ubuntu_20.04}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2021-02-18T00:29:55.309 INFO:scrape:
2021-02-18T00:29:55.310 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735bf90a2cfdf28b8ab4913090a3aedebc37463c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2021-02-18T00:29:55.310 INFO:scrape:1 jobs: ['5890904']
2021-02-18T00:29:55.311 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}']
2021-02-18T00:29:55.311 INFO:scrape:
2021-02-18T00:29:55.311 INFO:scrape:Failure: "2021-02-17T10:28:42.812572+0000 mon.a (mon.0) 22 : cluster [WRN] Health check failed: 1/21 mons down, quorum a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,r,s,t,u (MON_DOWN)" in cluster log
2021-02-18T00:29:55.312 INFO:scrape:1 jobs: ['5890633']
2021-02-18T00:29:55.312 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/multimon/{clusters/21', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_no_skews}']
2021-02-18T00:29:55.313 INFO:scrape:
2021-02-18T00:29:55.313 INFO:scrape:Failure: Command failed on smithi067 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:735bf90a2cfdf28b8ab4913090a3aedebc37463c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6fa8f6a-7105-11eb-9009-001a4aab830c -- ceph orch daemon add mon 'smithi067:[v1:172.21.15.67:6789]=b'"
2021-02-18T00:29:55.313 INFO:scrape:1 jobs: ['5890551']
2021-02-18T00:29:55.314 INFO:scrape:suites: ['1-install/luminous', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/few', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}']
2021-02-18T00:29:55.314 INFO:scrape:
2021-02-18T00:29:55.315 INFO:scrape:Crash: "2021-02-17T10:35:34.464311+0000 osd.0 (osd.0) 47 : cluster [WRN] Monitor daemon marked osd.0 down, but it is still running" in cluster log
2021-02-18T00:29:55.315 INFO:scrape:ceph version 17.0.0-827-g735bf90a (735bf90a2cfdf28b8ab4913090a3aedebc37463c) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9c9cbe4b20]
 2: pthread_cond_timedwait()
 3: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x355) [0x55a2b5e03e95]
 4: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55a2b646a794]
 5: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a2b646d434]
 6: (Thread::_entry_func(void*)+0xd) [0x55a2b645c29d]
 7: /lib64/libpthread.so.0(+0x814a) [0x7f9c9cbda14a]
 8: clone()
2021-02-18T00:29:55.315 INFO:scrape:1 jobs: ['5890620']
2021-02-18T00:29:55.316 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/lost-unfound-delete', 'supported-random-distro$/{rhel_8}}']
2021-02-18T00:29:55.316 INFO:scrape: