2021-02-23T18:12:04.240 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-23T18:12:04.246 INFO:teuthology.results:subset: '1/115000' 2021-02-23T18:12:04.246 INFO:teuthology.results:seed: '2477' 2021-02-23T21:59:01.317 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-23T21:59:01.771 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-24T09:31:26.266 INFO:teuthology.results:Tests finished! gathering results... 2021-02-24T09:40:24.408 INFO:scrape:Found 361 jobs 2021-02-24T09:40:37.918 INFO:scrape:Found 9 distinct failure reasons 2021-02-24T09:40:37.924 INFO:scrape:Failure: ceph version 16.1.0-342-g9ed1c344-1bionic was not installed, found 16.1.0-340-g71ca6256-1bionic. 2021-02-24T09:40:37.924 INFO:scrape:1 jobs: ['5906080'] 2021-02-24T09:40:37.925 INFO:scrape:suites: ['1-install/pacific', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2021-02-24T09:40:37.925 INFO:scrape: 2021-02-24T09:40:37.926 INFO:scrape:Crash: Command failed (workunit test rados/test_rados_tool.sh) on smithi014 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ad453be029e5c2348acbbbf6cf28bf836201d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_rados_tool.sh' 2021-02-24T09:40:37.926 INFO:scrape:ceph version 17.0.0-1008-g61ad453b (61ad453be029e5c2348acbbbf6cf28bf836201d1) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f67b3aafb20] 2: gsignal() 3: abort() 4: /usr/bin/rados(+0x4b838) [0x555786821838] 5: main() 6: __libc_start_main() 7: _start() 2021-02-24T09:40:37.927 INFO:scrape:2 jobs: ['5906058', '5906099'] 2021-02-24T09:40:37.927 INFO:scrape:suites intersection: [] 2021-02-24T09:40:37.927 INFO:scrape:suites union: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/radostool', 'rados/standalone/{mon_election/classic', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'workloads/misc}'] 2021-02-24T09:40:37.928 INFO:scrape: 2021-02-24T09:40:37.928 INFO:scrape:Dead: b'2021-02-24T09:27:20.550 DEBUG:teuthology.exit:Finished running handlers' 2021-02-24T09:40:37.929 INFO:scrape:1 jobs: ['5906135'] 2021-02-24T09:40:37.929 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] 2021-02-24T09:40:37.930 INFO:scrape: 2021-02-24T09:40:37.930 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi161 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ad453be029e5c2348acbbbf6cf28bf836201d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2021-02-24T09:40:37.931 INFO:scrape:4 jobs: ['5905913', '5906172', '5906000', '5906088'] 2021-02-24T09:40:37.931 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool'] 2021-02-24T09:40:37.931 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2021-02-24T09:40:37.932 INFO:scrape: 2021-02-24T09:40:37.932 INFO:scrape:Failure: rgw multisite test failures 2021-02-24T09:40:37.933 INFO:scrape:1 jobs: ['5905930'] 2021-02-24T09:40:37.933 INFO:scrape:suites: ['frontend', 'overrides', 'rados/upgrade/pacific-x/rgw-multisite/{clusters', 'realm', 'tasks', 'upgrade/primary}'] 2021-02-24T09:40:37.934 INFO:scrape: 2021-02-24T09:40:37.934 INFO:scrape:Failure: Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 714c88d4-7618-11eb-902c-001a4aab830c -e sha1=61ad453be029e5c2348acbbbf6cf28bf836201d1 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2021-02-24T09:40:37.935 INFO:scrape:2 jobs: ['5906025', '5905845'] 2021-02-24T09:40:37.935 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-24T09:40:37.936 INFO:scrape:suites union: ['1-tasks', 'distro1$/{centos_8.0}', 'distro1$/{centos_latest}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-24T09:40:37.936 INFO:scrape: 2021-02-24T09:40:37.937 INFO:scrape:Failure: Test failure: test_host_devices (tasks.mgr.dashboard.test_host.HostControllerTest) 2021-02-24T09:40:37.937 INFO:scrape:2 jobs: ['5906001', '5906180'] 2021-02-24T09:40:37.937 INFO:scrape:suites intersection: ['debug/mgr', 'objectstore/filestore-xfs', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/dashboard}'] 2021-02-24T09:40:37.938 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/filestore-xfs', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/dashboard}'] 2021-02-24T09:40:37.938 INFO:scrape: 2021-02-24T09:40:37.955 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ad453be029e5c2348acbbbf6cf28bf836201d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2021-02-24T09:40:37.955 INFO:scrape:1 jobs: ['5906184'] 2021-02-24T09:40:37.956 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] 2021-02-24T09:40:37.956 INFO:scrape: 2021-02-24T09:40:37.956 INFO:scrape:Failure: Test failure: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) 2021-02-24T09:40:37.957 INFO:scrape:1 jobs: ['5905923'] 2021-02-24T09:40:37.957 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'task/test_orch_cli}'] 2021-02-24T09:40:37.958 INFO:scrape: