2021-05-16T01:25:31.639 INFO:root:teuthology version: 1.1.0-cc3bac84 2021-05-16T01:25:31.641 INFO:teuthology.results:seed: '2724' 2021-05-16T01:25:33.063 INFO:root:teuthology version: 1.1.0-cc3bac84 2021-05-16T01:25:33.065 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-16T13:26:18.597 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-05-16T13:26:19.134 INFO:scrape:Found 5 jobs 2021-05-16T13:26:19.142 INFO:scrape:Found 3 distinct failure reasons 2021-05-16T13:26:19.143 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds 2021-05-16T13:26:19.143 INFO:scrape:1 jobs: ['6116417'] 2021-05-16T13:26:19.144 INFO:scrape:suites: ['1-ceph-install/pacific.', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack'] 2021-05-16T13:26:19.144 INFO:scrape: 2021-05-16T13:26:19.145 INFO:scrape:Dead: b'2021-05-16T13:26:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2021-05-16T13:26:18.978+0000 7f35da44d700 -1 WARNING: all dangerous and experimental features are enabled.' 2021-05-16T13:26:19.145 INFO:scrape:1 jobs: ['6116414'] 2021-05-16T13:26:19.146 INFO:scrape:suites: ['1-ceph-install/pacific.', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack'] 2021-05-16T13:26:19.146 INFO:scrape: 2021-05-16T13:26:19.147 INFO:scrape:Failure: "2021-05-16T02:30:28.805614+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2021-05-16T13:26:19.147 INFO:scrape:1 jobs: ['6116413'] 2021-05-16T13:26:19.147 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade'] 2021-05-16T13:26:19.148 INFO:scrape: 2021-05-16T13:26:19.149 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/ failed: 2 dead: 0 running: 1 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6116413] upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 01:14:21 info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116413/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116413/ "2021-05-16T02:30:28.805614+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [6116417] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific. 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 02:21:14 info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116417/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=493e254947a74cf6889818f3e59ec2fc reached maximum tries (500) after waiting for 3000 seconds Running ================================================================= [6116414] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific. 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116414/ Pass ================================================================= [6116415] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific. 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} time: 04:24:51 info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116415/ [6116416] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific. 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} time: 05:47:12 info: http://pulpito.ceph.com/teuthology-2021-05-16_01:25:02-upgrade:pacific-p2p-pacific-distro-basic-smithi/6116416/