2021-03-14T15:14:26.324 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-14T15:14:26.326 INFO:teuthology.results:subset: '111/999' 2021-03-14T15:14:26.327 INFO:teuthology.results:seed: '5355' 2021-03-14T16:16:36.640 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-14T16:16:36.677 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-15T04:17:30.758 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-03-15T04:18:01.435 INFO:scrape:Found 56 jobs 2021-03-15T04:18:47.241 INFO:scrape:Found 11 distinct failure reasons 2021-03-15T04:18:47.243 INFO:scrape:Failure: {'smithi170.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi170.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi170.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}, 'smithi125.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi125.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi125.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}} 2021-03-15T04:18:47.244 INFO:scrape:1 jobs: ['5965216'] 2021-03-15T04:18:47.244 INFO:scrape:suites: ['fixed-2', 'mon_election/connectivity', 'rados:cephadm/smoke/{distro/ubuntu_latest', 'start}'] 2021-03-15T04:18:47.245 INFO:scrape: 2021-03-15T04:18:47.246 INFO:scrape:Failure: timeout expired in wait_until_healthy 2021-03-15T04:18:47.246 INFO:scrape:14 jobs: ['5965200', '5965190', '5965199', '5965214', '5965231', '5965192', '5965196', '5965233', '5965187', '5965184', '5965224', '5965208', '5965203', '5965220'] 2021-03-15T04:18:47.247 INFO:scrape:suites intersection: ['fixed-2'] 2021-03-15T04:18:47.248 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados:cephadm/smoke/{distro/centos_8.0', 'rados:cephadm/smoke/{distro/centos_latest', 'rados:cephadm/smoke/{distro/rhel_8.0', 'rados:cephadm/smoke/{distro/rhel_latest', 'rados:cephadm/smoke/{distro/ubuntu_18.04', 'rados:cephadm/smoke/{distro/ubuntu_latest', 'rados:cephadm/with-work/{distro/centos_8.0', 'rados:cephadm/with-work/{distro/centos_latest', 'rados:cephadm/with-work/{distro/rhel_8.0', 'rados:cephadm/with-work/{distro/rhel_latest', 'rados:cephadm/with-work/{distro/ubuntu_18.04', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}'] 2021-03-15T04:18:47.248 INFO:scrape: 2021-03-15T04:18:47.248 INFO:scrape:Failure: SSH connection to smithi125 was lost: 'rpm -q kernel --last | head -n 1' 2021-03-15T04:18:47.249 INFO:scrape:1 jobs: ['5965226'] 2021-03-15T04:18:47.249 INFO:scrape:suites: ['fixed-2', 'mon_election/connectivity', 'rados:cephadm/smoke/{distro/centos_latest', 'start}'] 2021-03-15T04:18:47.250 INFO:scrape: 2021-03-15T04:18:47.250 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi012 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2021-03-15T04:18:47.251 INFO:scrape:1 jobs: ['5965218'] 2021-03-15T04:18:47.251 INFO:scrape:suites: ['rados:cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] 2021-03-15T04:18:47.252 INFO:scrape: 2021-03-15T04:18:47.252 INFO:scrape:Dead: b'2021-03-15T04:18:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Mar 15 04:18:02 smithi111 bash[14576]: cluster 2021-03-15T04:18:01.033177+0000 mgr.x (mgr.34520) 25976 : cluster [DBG] pgmap v21136: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail' 2021-03-15T04:18:47.253 INFO:scrape:1 jobs: ['5965235'] 2021-03-15T04:18:47.253 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_latest}', 'fixed-2', 'mon_election/connectivity}', 'rados:cephadm/upgrade/{1-start'] 2021-03-15T04:18:47.253 INFO:scrape: 2021-03-15T04:18:47.254 INFO:scrape:Failure: Command failed on smithi125 with status 1: 'sudo yum install -y kernel' 2021-03-15T04:18:47.256 INFO:scrape:8 jobs: ['5965222', '5965207', '5965185', '5965211', '5965182', '5965215', '5965204', '5965221'] 2021-03-15T04:18:47.256 INFO:scrape:suites intersection: [] 2021-03-15T04:18:47.257 INFO:scrape:suites union: ['1-install/nautilus-v2only', '2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'rados', 'rados:cephadm/smoke-roleless/{distro/centos_8.0', 'rados:cephadm/smoke-roleless/{distro/ubuntu_18.04_podman', 'rados:cephadm/smoke/{distro/ubuntu_18.04_podman', 'rados:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados:cephadm/upgrade/{1-start', 'rados:cephadm/with-work/{distro/ubuntu_18.04_podman', 'rados:cephadm/workunits/{distro/ubuntu_18.04_podman', 'start', 'start}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}'] 2021-03-15T04:18:47.257 INFO:scrape: 2021-03-15T04:18:47.258 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2021-03-15T04:18:47.258 INFO:scrape:2 jobs: ['5965194', '5965236'] 2021-03-15T04:18:47.259 INFO:scrape:suites intersection: ['rados:cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2021-03-15T04:18:47.259 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados:cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2021-03-15T04:18:47.259 INFO:scrape: 2021-03-15T04:18:47.260 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2021-03-15T04:18:47.260 INFO:scrape:1 jobs: ['5965225'] 2021-03-15T04:18:47.261 INFO:scrape:suites: ['mon_election/connectivity', 'rados:cephadm/workunits/{distro/centos_latest', 'task/test_orch_cli}'] 2021-03-15T04:18:47.261 INFO:scrape: 2021-03-15T04:18:47.262 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2021-03-15T04:18:47.262 INFO:scrape:2 jobs: ['5965193', '5965206'] 2021-03-15T04:18:47.263 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'fixed-2', 'rados:cephadm/upgrade/{1-start'] 2021-03-15T04:18:47.263 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados:cephadm/upgrade/{1-start'] 2021-03-15T04:18:47.263 INFO:scrape: 2021-03-15T04:18:47.264 INFO:scrape:Failure: Command failed on smithi060 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:6a19e303187c2defceb9c785284ca401a4309c47 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b93d914e-84da-11eb-908e-001a4aab830c -- ceph orch daemon add mon smithi060:172.21.15.60=smithi060' 2021-03-15T04:18:47.265 INFO:scrape:1 jobs: ['5965188'] 2021-03-15T04:18:47.265 INFO:scrape:suites: ['rados:cephadm/smoke-roleless/{distro/rhel_latest', 'start}'] 2021-03-15T04:18:47.265 INFO:scrape: 2021-03-15T04:18:47.266 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2021-03-15T04:18:47.266 INFO:scrape:1 jobs: ['5965202'] 2021-03-15T04:18:47.267 INFO:scrape:suites: ['mon_election/connectivity', 'rados:cephadm/workunits/{distro/centos_latest', 'task/test_cephadm}'] 2021-03-15T04:18:47.267 INFO:scrape: