2020-12-04T14:24:03.107 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-04T14:24:03.109 INFO:teuthology.results:seed: '7819' 2020-12-04T14:33:47.013 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-04T14:33:47.972 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-04T22:26:56.061 INFO:teuthology.results:Tests finished! gathering results... 2020-12-04T22:26:58.870 INFO:scrape:Found 16 jobs 2020-12-04T22:26:58.877 INFO:scrape:Found 1 distinct failure reasons 2020-12-04T22:26:58.877 INFO:scrape:Failure: "2020-12-04T14:55:07.648604+0000 mon.c (mon.0) 267 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log 2020-12-04T22:26:58.878 INFO:scrape:7 jobs: ['5680625', '5680624', '5680632', '5680621', '5680620', '5680628', '5680633'] 2020-12-04T22:26:58.878 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_latest}'] 2020-12-04T22:26:58.879 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2020-12-04T22:26:58.879 INFO:scrape: 2020-12-04T22:26:58.881 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [5680620] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:30:28 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680620/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680620/ "2020-12-04T14:49:12.370517+0000 mon.c (mon.0) 265 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680621] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 02:17:00 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680621/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680621/ "2020-12-04T14:50:08.838067+0000 mon.c (mon.1) 120 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680624] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:35:03 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680624/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680624/ "2020-12-04T14:56:27.881933+0000 mon.c (mon.0) 276 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680625] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:42:09 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680625/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680625/ "2020-12-04T14:55:07.648604+0000 mon.c (mon.0) 267 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680628] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:45:24 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680628/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680628/ "2020-12-04T18:11:58.915500+0000 mon.c (mon.0) 268 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680632] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:27:10 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680632/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680632/ "2020-12-04T19:10:08.713991+0000 mon.c (mon.0) 279 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum c,a" in cluster log [5680633] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:57:06 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680633/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680633/ "2020-12-04T15:20:00.000146+0000 mon.c (mon.0) 1901 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log Pass ================================================================= [5680619] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:44:57 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680619/ [5680622] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:36:20 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680622/ [5680623] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:41:49 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680623/ [5680626] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:34:28 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680626/ [5680627] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:41:35 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680627/ [5680629] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:04:12 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680629/ [5680630] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:37:28 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680630/ [5680631] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:40:46 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680631/ [5680634] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:37:28 info: http://pulpito.ceph.com/teuthology-2020-12-04_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5680634/