2020-11-28T22:46:58.531 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-28T22:46:58.533 INFO:teuthology.results:seed: '3709' 2020-11-28T22:58:23.932 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-28T22:58:23.969 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-29T03:34:28.791 INFO:teuthology.results:Tests finished! gathering results... 2020-11-29T03:34:31.385 INFO:scrape:Found 16 jobs 2020-11-29T03:34:32.959 INFO:scrape:Found 3 distinct failure reasons 2020-11-29T03:34:32.960 INFO:scrape:Failure: failed to recover before timeout expired 2020-11-29T03:34:32.961 INFO:scrape:1 jobs: ['5657526'] 2020-11-29T03:34:32.961 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-11-29T03:34:32.962 INFO:scrape: 2020-11-29T03:34:32.962 INFO:scrape:Failure: "2020-11-28 23:49:25.217725 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi018:x (7500), after 304.382 seconds" in cluster log 2020-11-29T03:34:32.962 INFO:scrape:1 jobs: ['5657523'] 2020-11-29T03:34:32.963 INFO:scrape:suites: ['supported-all-distro/rhel_7}', 'upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade'] 2020-11-29T03:34:32.963 INFO:scrape: 2020-11-29T03:34:32.964 INFO:scrape:Failure: timed out waiting for admin_socket to appear after osd.3 restart 2020-11-29T03:34:32.964 INFO:scrape:1 jobs: ['5657519'] 2020-11-29T03:34:32.964 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '7-final-workload/{rbd-python', 'objectstore/default', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/rhel_7', 'thrashosds-health}', 'upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack'] 2020-11-29T03:34:32.965 INFO:scrape: 2020-11-29T03:34:32.966 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [5657519] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/rhel_7 thrashosds-health} ----------------------------------------------------------------- time: 02:50:11 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657519/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96bd3356920e437fa0dc7ac1028b79fb timed out waiting for admin_socket to appear after osd.3 restart [5657523] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/rhel_7} ----------------------------------------------------------------- time: 01:15:43 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657523/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657523/ "2020-11-28 23:49:25.217725 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi018:x (7500), after 304.382 seconds" in cluster log [5657526] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/centos_latest thrashosds-health} ----------------------------------------------------------------- time: 02:36:24 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657526/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657526/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6aa3c0ff1d340f4a7392debd995fba8 failed to recover before timeout expired Pass ================================================================= [5657515] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_latest} time: 01:16:16 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657515/ [5657517] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health} time: 03:59:14 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657517/ [5657521] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:20:34 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657521/ [5657525] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:02:41 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657525/ [5657527] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health} time: 03:10:14 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657527/ [5657528] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_16.04} time: 01:11:05 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657528/ [5657529] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:56:44 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657529/ [5657530] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_latest thrashosds-health} time: 03:19:19 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657530/ [5657531] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health} time: 03:22:31 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657531/ [5657532] upgrade:nautilus-p2p/nautilus-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} time: 01:08:47 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657532/ [5657533] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/rhel_7 thrashosds-health} time: 03:57:04 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657533/ [5657534] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_16.04 thrashosds-health} time: 03:16:18 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657534/ [5657535] upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} time: 03:10:38 info: http://pulpito.ceph.com/teuthology-2020-11-26_01:25:03-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5657535/