2020-08-27T20:34:08.275 INFO:root:teuthology version: 1.0.0-8271c95 2020-08-27T20:34:08.277 INFO:teuthology.results:seed: '6670' 2020-08-27T20:47:57.639 INFO:root:teuthology version: 1.0.0-8271c95 2020-08-27T20:47:57.668 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-28T02:17:05.219 INFO:teuthology.results:Tests finished! gathering results... 2020-08-28T02:17:09.685 INFO:scrape:Found 16 jobs 2020-08-28T02:17:11.806 INFO:scrape:Found 2 distinct failure reasons 2020-08-28T02:17:11.807 INFO:scrape:Failure: "2020-08-27 21:59:10.239731 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi022:x (8268), after 304.704 seconds" in cluster log 2020-08-28T02:17:11.808 INFO:scrape:2 jobs: ['5379279', '5379275'] 2020-08-28T02:17:11.808 INFO:scrape:suites intersection: ['upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-08-28T02:17:11.809 INFO:scrape:suites union: ['supported-all-distro/rhel_7}', 'supported-all-distro/ubuntu_16.04}', 'upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-08-28T02:17:11.809 INFO:scrape: 2020-08-28T02:17:11.810 INFO:scrape:Failure: timed out waiting for admin_socket to appear after osd.2 restart 2020-08-28T02:17:11.810 INFO:scrape:3 jobs: ['5379273', '5379281', '5379277'] 2020-08-28T02:17:11.810 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-08-28T02:17:11.811 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'supported-all-distro/rhel_7', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-08-28T02:17:11.812 INFO:scrape: 2020-08-28T02:17:11.814 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [5379273] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/rhel_7 thrashosds-health} ----------------------------------------------------------------- time: 02:52:20 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379273/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379273/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=77a3e0cfff904a5fa583236fd1a0bc51 timed out waiting for admin_socket to appear after osd.2 restart [5379275] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/rhel_7} ----------------------------------------------------------------- time: 01:20:22 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379275/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379275/ "2020-08-27 22:07:09.049970 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi082:x (7641), after 301.477 seconds" in cluster log [5379277] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/centos_latest thrashosds-health} ----------------------------------------------------------------- time: 03:08:58 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379277/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379277/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=c21fd489031e4ff9a5e4d24a2fe2ca05 timed out waiting for admin_socket to appear after osd.0 restart [5379279] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_16.04} ----------------------------------------------------------------- time: 01:16:45 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379279/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379279/ "2020-08-27 21:59:10.239731 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi022:x (8268), after 304.704 seconds" in cluster log [5379281] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 02:06:10 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379281/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379281/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=912d55436b9a4d0ba1646abe5f305846 timed out waiting for admin_socket to appear after osd.3 restart Pass ================================================================= [5379270] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_latest} time: 01:21:20 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379270/ [5379272] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health} time: 04:10:08 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379272/ [5379274] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:57:19 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379274/ [5379276] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:18:34 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379276/ [5379278] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health} time: 03:05:01 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379278/ [5379280] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health} time: 04:02:10 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379280/ [5379282] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health} time: 03:36:29 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379282/ [5379283] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} time: 01:15:44 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379283/ [5379284] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/rhel_7 thrashosds-health} time: 04:10:02 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379284/ [5379285] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:31:16 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379285/ [5379286] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} time: 03:30:16 info: http://pulpito.ceph.com/teuthology-2020-08-27_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5379286/