2020-12-02T17:17:46.991 INFO:root:teuthology version: 1.0.0-e6b33b7b
2020-12-02T17:17:46.993 INFO:teuthology.results:subset: '0/9999'
2020-12-02T17:17:46.993 INFO:teuthology.results:seed: '669'
2020-12-02T23:04:53.472 INFO:root:teuthology version: 1.0.0-e6b33b7b
2020-12-02T23:04:53.510 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-12-03T10:21:14.040 INFO:teuthology.results:Tests finished! gathering results...
2020-12-03T10:21:56.004 INFO:scrape:Found 250 jobs
2020-12-03T10:22:15.263 INFO:scrape:Found 10 distinct failure reasons
2020-12-03T10:22:15.265 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bb5d201906bb62bfeaa17ffe33e9a1c6bb391479 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
2020-12-03T10:22:15.265 INFO:scrape:2 jobs: ['5675422', '5675575']
2020-12-03T10:22:15.266 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/centos_latest', 'task/test_adoption}']
2020-12-03T10:22:15.266 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'task/test_adoption}']
2020-12-03T10:22:15.267 INFO:scrape:
2020-12-03T10:22:15.267 INFO:scrape:Dead: b'2020-12-03T10:18:30.784 DEBUG:teuthology.exit:Finished running handlers'
2020-12-03T10:22:15.268 INFO:scrape:2 jobs: ['5675573', '5675522']
2020-12-03T10:22:15.268 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
2020-12-03T10:22:15.268 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']
2020-12-03T10:22:15.269 INFO:scrape:
2020-12-03T10:22:15.269 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bb5d201906bb62bfeaa17ffe33e9a1c6bb391479 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2020-12-03T10:22:15.270 INFO:scrape:2 jobs: ['5675608', '5675451']
2020-12-03T10:22:15.270 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
2020-12-03T10:22:15.271 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
2020-12-03T10:22:15.271 INFO:scrape:
2020-12-03T10:22:15.271 INFO:scrape:Failure: Command failed on smithi170 with status 5: 'sudo systemctl stop ceph-14e0fb22-34e4-11eb-980d-001a4aab830c@mon.a'
2020-12-03T10:22:15.272 INFO:scrape:43 jobs
2020-12-03T10:22:15.273 INFO:scrape:suites intersection: []
2020-12-03T10:22:15.273 INFO:scrape:suites union: ['1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/mimic-v1only', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados', 'rados/cephadm/dashboard/{distro/centos_latest', 'rados/cephadm/smoke-roleless/{distro/centos_8.0', 'rados/cephadm/smoke-roleless/{distro/centos_latest', 'rados/cephadm/smoke-roleless/{distro/rhel_8.0', 'rados/cephadm/smoke-roleless/{distro/rhel_latest', 'rados/cephadm/smoke-roleless/{distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman', 'rados/cephadm/smoke-roleless/{distro/ubuntu_latest', 'rados/cephadm/smoke/{distro/centos_8.0', 'rados/cephadm/smoke/{distro/centos_latest', 'rados/cephadm/smoke/{distro/rhel_8.0', 'rados/cephadm/smoke/{distro/rhel_latest', 'rados/cephadm/smoke/{distro/ubuntu_18.04', 'rados/cephadm/smoke/{distro/ubuntu_18.04_podman', 'rados/cephadm/smoke/{distro/ubuntu_latest', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/cephadm/with-work/{distro/centos_8.0', 'rados/cephadm/with-work/{distro/centos_latest', 'rados/cephadm/with-work/{distro/rhel_8.0', 'rados/cephadm/with-work/{distro/rhel_latest', 'rados/cephadm/with-work/{distro/ubuntu_18.04', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'start', 'start}', 'task/test_e2e}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}']
2020-12-03T10:22:15.274 INFO:scrape:
2020-12-03T10:22:15.274 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=21174)
2020-12-03T10:22:15.275 INFO:scrape:1 jobs: ['5675462']
2020-12-03T10:22:15.275 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']
2020-12-03T10:22:15.276 INFO:scrape:
2020-12-03T10:22:15.276 INFO:scrape:Failure: "2020-12-02T18:12:42.878653+0000 mon.b (mon.0) 22 : cluster [WRN] Health check failed: 1/21 mons down, quorum b,a,c,e,d,f,h,g,i,k,j,l,n,m,o,q,p,r,t,s (MON_DOWN)" in cluster log
2020-12-03T10:22:15.276 INFO:scrape:2 jobs: ['5675386', '5675537']
2020-12-03T10:22:15.277 INFO:scrape:suites intersection: ['rados']
2020-12-03T10:22:15.277 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/multimon/{clusters/21', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/mon_clock_no_skews}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect}']
2020-12-03T10:22:15.278 INFO:scrape:
2020-12-03T10:22:15.278 INFO:scrape:Failure: reached maximum tries (60) after waiting for 900 seconds
2020-12-03T10:22:15.279 INFO:scrape:2 jobs: ['5675609', '5675414']
2020-12-03T10:22:15.279 INFO:scrape:suites: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'rados/cephadm/upgrade/{1-start']
2020-12-03T10:22:15.279 INFO:scrape:
2020-12-03T10:22:15.280 INFO:scrape:Failure: Command failed (workunit test mon/mon-bind.sh) on smithi144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bb5d201906bb62bfeaa17ffe33e9a1c6bb391479 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh'
2020-12-03T10:22:15.280 INFO:scrape:1 jobs: ['5675518']
2020-12-03T10:22:15.281 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{ubuntu_latest}', 'workloads/mon}']
2020-12-03T10:22:15.281 INFO:scrape:
2020-12-03T10:22:15.281 INFO:scrape:Failure: Command failed (workunit test rados/test_python.sh) on smithi146 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bb5d201906bb62bfeaa17ffe33e9a1c6bb391479 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh'
2020-12-03T10:22:15.282 INFO:scrape:1 jobs: ['5675463']
2020-12-03T10:22:15.282 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}']
2020-12-03T10:22:15.283 INFO:scrape:
2020-12-03T10:22:15.284 INFO:scrape:Failure: "2020-12-02T22:05:45.611527+0000 osd.4 (osd.4) 487 : cluster [ERR] scrub 60.e 60:7243c543:test-rados-api-smithi013-32841-74::foo:24 : size 0 != clone_size 10" in cluster log
2020-12-03T10:22:15.284 INFO:scrape:2 jobs: ['5675528', '5675427']
2020-12-03T10:22:15.284 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health', 'workloads/rados_api_tests}']
2020-12-03T10:22:15.285 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/rados_api_tests}']
2020-12-03T10:22:15.285 INFO:scrape: