2020-10-02T17:27:09.240 INFO:root:teuthology version: 1.0.0-75888fc 2020-10-02T17:27:09.241 INFO:teuthology.results:seed: '6041' 2020-10-02T17:35:50.207 INFO:root:teuthology version: 1.0.0-75888fc 2020-10-02T17:35:50.248 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-02T21:55:54.917 INFO:teuthology.results:Tests finished! gathering results... 2020-10-02T21:55:57.235 INFO:scrape:Found 16 jobs 2020-10-02T21:55:58.256 INFO:scrape:Found 3 distinct failure reasons 2020-10-02T21:55:58.257 INFO:scrape:Failure: "2020-10-02 18:22:23.645905 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi089:x (7962), after 301.406 seconds" in cluster log 2020-10-02T21:55:58.257 INFO:scrape:1 jobs: ['5464277'] 2020-10-02T21:55:58.257 INFO:scrape:suites: ['supported-all-distro/ubuntu_16.04}', 'upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-10-02T21:55:58.258 INFO:scrape: 2020-10-02T21:55:58.258 INFO:scrape:Failure: timed out waiting for admin_socket to appear after osd.1 restart 2020-10-02T21:55:58.258 INFO:scrape:2 jobs: ['5464275', '5464283'] 2020-10-02T21:55:58.259 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-10-02T21:55:58.259 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'supported-all-distro/ubuntu_16.04', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-10-02T21:55:58.259 INFO:scrape: 2020-10-02T21:55:58.260 INFO:scrape:Failure: failed to recover before timeout expired 2020-10-02T21:55:58.260 INFO:scrape:1 jobs: ['5464271'] 2020-10-02T21:55:58.260 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/rhel_7', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-10-02T21:55:58.260 INFO:scrape: 2020-10-02T21:55:58.262 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [5464271] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/rhel_7 thrashosds-health} ----------------------------------------------------------------- time: 02:27:47 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464271/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464271/ sentry: https://sentry.ceph.com/sentry/sepia/?query=2e571cdcc27a423ebad8571b37739ea4 failed to recover before timeout expired [5464275] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/centos_latest thrashosds-health} ----------------------------------------------------------------- time: 02:58:38 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464275/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464275/ sentry: https://sentry.ceph.com/sentry/sepia/?query=d5cc6adf16b74c6ba7fa8a1d0e3cb1fe timed out waiting for admin_socket to appear after osd.1 restart [5464277] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_16.04} ----------------------------------------------------------------- time: 01:07:17 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464277/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464277/ "2020-10-02 18:22:23.645905 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi089:x (7962), after 301.406 seconds" in cluster log [5464283] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_16.04 thrashosds-health} ----------------------------------------------------------------- time: 02:29:54 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464283/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464283/ sentry: https://sentry.ceph.com/sentry/sepia/?query=8d63b4ddc1fc434dbc2925c8ec5673d4 timed out waiting for admin_socket to appear after osd.1 restart Pass ================================================================= [5464268] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_latest} time: 01:10:56 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464268/ [5464270] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health} time: 03:51:29 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464270/ [5464272] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_16.04 thrashosds-health} time: 02:48:05 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464272/ [5464273] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/rhel_7} time: 01:13:14 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464273/ [5464274] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 03:58:35 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464274/ [5464276] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health} time: 03:11:19 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464276/ [5464278] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:37:41 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464278/ [5464279] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_latest thrashosds-health} time: 03:18:32 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464279/ [5464280] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health} time: 03:43:57 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464280/ [5464281] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} time: 01:06:53 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464281/ [5464282] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/rhel_7 thrashosds-health} time: 03:51:08 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464282/ [5464284] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} time: 03:42:44 info: http://pulpito.ceph.com/teuthology-2020-09-24_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5464284/