2020-02-14T18:19:14.624 INFO:root:teuthology version: 1.0.0-ee19bca 2020-02-14T18:19:14.624 INFO:teuthology.results:subset: '100/9000' 2020-02-14T18:19:14.625 INFO:teuthology.results:seed: '3872' 2020-02-14T18:28:11.397 INFO:root:teuthology version: 1.0.0-ee19bca 2020-02-14T18:28:11.406 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-02-15T06:28:27.709 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-02-15T06:28:27.794 INFO:teuthology.results:Sending results to dzafman@redhat.com: Test Run: dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/ failed: 2 dead: 3 running: 1 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [4764131] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/rhel_7.yaml thrashosds-health.yaml} ----------------------------------------------------------------- time: 04:57:41 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764131/ log: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764131/ Command failed on smithi037 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats' [4764134] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml supported-all-distro/ubuntu_16.04.yaml thrashosds-health.yaml} ----------------------------------------------------------------- time: 06:05:15 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764134/ log: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764134/ global name 'stdout' is not defined Dead ================================================================= [4764140] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/centos_latest.yaml thrashosds-health.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764140/ log: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764140/ [4764143] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml supported-all-distro/rhel_7.yaml thrashosds-health.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764143/ log: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764143/ [4764144] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/filestore-xfs.yaml supported-all-distro/rhel_7.yaml thrashosds-health.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764144/ log: http://qa-proxy.ceph.com/teuthology/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764144/ Running ================================================================= [4764146] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/ubuntu_16.04.yaml thrashosds-health.yaml} info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764146/ Pass ================================================================= [4764138] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/bluestore-bitmap.yaml supported-all-distro/ubuntu_latest.yaml thrashosds-health.yaml} time: 00:45:31 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764138/ [4764139] upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/centos_latest.yaml} time: 02:13:26 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764139/ [4764142] upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/filestore-xfs.yaml supported-all-distro/rhel_7.yaml} time: 02:26:29 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764142/ [4764145] upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/ubuntu_16.04.yaml} time: 02:11:13 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764145/ [4764132] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/bluestore-bitmap.yaml supported-all-distro/rhel_7.yaml thrashosds-health.yaml} time: 00:54:49 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764132/ [4764133] upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/filestore-xfs.yaml supported-all-distro/ubuntu_16.04.yaml} time: 02:11:25 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764133/ [4764135] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/filestore-xfs.yaml supported-all-distro/ubuntu_16.04.yaml thrashosds-health.yaml} time: 00:43:28 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764135/ [4764136] upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/ubuntu_latest.yaml} time: 02:13:38 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764136/ [4764137] upgrade:mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml rgw-swift-ragweed_check.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml supported-all-distro/ubuntu_latest.yaml thrashosds-health.yaml} time: 05:09:19 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764137/ [4764141] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/bluestore-bitmap.yaml supported-all-distro/centos_latest.yaml thrashosds-health.yaml} time: 00:48:29 info: http://pulpito.ceph.com/dzafman-2020-02-14_10:17:39-upgrade:mimic-x-master-distro-basic-smithi/4764141/ 2020-02-15T06:28:27.871 INFO:teuthology.results:starting coverage generation