2020-10-17T17:51:38.390 INFO:root:teuthology version: 1.0.0-b106d09 2020-10-17T17:51:38.391 INFO:teuthology.results:seed: '9371' 2020-10-17T18:13:17.416 INFO:root:teuthology version: 1.0.0-b106d09 2020-10-17T18:13:17.471 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-17T23:14:22.746 INFO:teuthology.results:Tests finished! gathering results... 2020-10-17T23:14:24.783 INFO:scrape:Found 16 jobs 2020-10-17T23:14:26.682 INFO:scrape:Found 2 distinct failure reasons 2020-10-17T23:14:26.683 INFO:scrape:Failure: failed to recover before timeout expired 2020-10-17T23:14:26.683 INFO:scrape:1 jobs: ['5525962'] 2020-10-17T23:14:26.684 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/rhel_7', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-10-17T23:14:26.684 INFO:scrape: 2020-10-17T23:14:26.684 INFO:scrape:Failure: "2020-10-17 18:46:28.488397 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi023:x (7524), after 300.571 seconds" in cluster log 2020-10-17T23:14:26.685 INFO:scrape:2 jobs: ['5525964', '5525972'] 2020-10-17T23:14:26.685 INFO:scrape:suites intersection: ['upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-10-17T23:14:26.685 INFO:scrape:suites union: ['supported-all-distro/rhel_7}', 'supported-all-distro/ubuntu_latest}', 'upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-10-17T23:14:26.686 INFO:scrape: 2020-10-17T23:14:26.687 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [5525962] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/rhel_7 thrashosds-health} ----------------------------------------------------------------- time: 03:08:19 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525962/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525962/ sentry: https://sentry.ceph.com/sentry/sepia/?query=19483be7e56d450ab2bfcb63cdabe6a0 failed to recover before timeout expired [5525964] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/rhel_7} ----------------------------------------------------------------- time: 01:13:39 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525964/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525964/ "2020-10-17 18:46:28.488397 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi023:x (7524), after 300.571 seconds" in cluster log [5525972] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 01:07:48 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525972/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525972/ "2020-10-17 19:56:44.889112 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi155:x (7623), after 300.091 seconds" in cluster log Pass ================================================================= [5525959] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_latest} time: 01:15:30 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525959/ [5525961] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health} time: 03:45:13 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525961/ [5525963] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_16.04 thrashosds-health} time: 04:17:35 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525963/ [5525965] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:07:58 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525965/ [5525966] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/centos_latest thrashosds-health} time: 03:31:12 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525966/ [5525967] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health} time: 03:18:56 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525967/ [5525968] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_16.04} time: 01:13:38 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525968/ [5525969] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:49:31 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525969/ [5525970] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_latest thrashosds-health} time: 03:40:25 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525970/ [5525971] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health} time: 03:16:54 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525971/ [5525973] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/rhel_7 thrashosds-health} time: 03:55:29 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525973/ [5525974] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:28:01 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525974/ [5525975] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} time: 03:25:45 info: http://pulpito.ceph.com/teuthology-2020-10-15_01:25:02-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5525975/