2020-12-10T16:16:50.749 INFO:root:teuthology version: 1.0.0-5f33661d 2020-12-10T16:16:50.752 INFO:teuthology.results:seed: '3576' 2020-12-10T16:28:46.030 INFO:root:teuthology version: 1.0.0-5f33661d 2020-12-10T16:28:46.033 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-11T01:55:55.675 INFO:teuthology.results:Tests finished! gathering results... 2020-12-11T01:55:58.946 INFO:scrape:Found 16 jobs 2020-12-11T01:55:58.955 INFO:scrape:Found 1 distinct failure reasons 2020-12-11T01:55:58.955 INFO:scrape:Failure: "2020-12-10T21:56:51.664776+0000 mon.c (mon.1) 115 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log 2020-12-11T01:55:58.956 INFO:scrape:2 jobs: ['5698719', '5698727'] 2020-12-11T01:55:58.956 INFO:scrape:suites intersection: ['1-nautilus-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-octopus', '7-final-workload', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2020-12-11T01:55:58.957 INFO:scrape:suites union: ['1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-octopus', '7-final-workload', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2020-12-11T01:55:58.957 INFO:scrape: 2020-12-11T01:55:58.958 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [5698719] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:43:22 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698719/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698719/ "2020-12-10T21:56:51.664776+0000 mon.c (mon.1) 115 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5698727] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:59:11 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698727/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698727/ "2020-12-10T22:59:45.235195+0000 mon.c (mon.0) 276 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log Pass ================================================================= [5698713] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:47:32 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698713/ [5698714] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:31:14 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698714/ [5698715] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:10:22 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698715/ [5698716] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:41:23 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698716/ [5698717] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:45:02 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698717/ [5698718] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:20:06 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698718/ [5698720] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:40:32 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698720/ [5698721] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:45:25 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698721/ [5698722] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:55:45 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698722/ [5698723] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:03:44 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698723/ [5698724] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:41:49 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698724/ [5698725] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:45:41 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698725/ [5698726] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:13:35 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698726/ [5698728] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:42:02 info: http://pulpito.ceph.com/teuthology-2020-12-10_16:13:48-upgrade:nautilus-x-octopus-distro-basic-smithi/5698728/