2021-06-23T06:36:33.020 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-23T06:36:33.022 INFO:teuthology.results:seed: '8752' 2021-06-23T06:52:34.245 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-23T06:52:34.281 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-06-23T07:05:35.181 INFO:teuthology.results:Tests finished! gathering results... 2021-06-23T07:05:35.529 INFO:scrape:Found 16 jobs 2021-06-23T07:05:35.530 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147985/teuthology.log 2021-06-23T07:05:35.907 INFO:scrape:Found 2 distinct failure reasons 2021-06-23T07:05:35.908 INFO:scrape:Dead: None 2021-06-23T07:05:35.909 INFO:scrape:13 jobs: ['6147985', '6147987', '6147986', '6147993', '6147997', '6148000', '6147990', '6147989', '6147991', '6147999', '6147994', '6147995', '6147998'] 2021-06-23T07:05:35.910 INFO:scrape: 2021-06-23T07:05:35.911 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=aa3f3e3ca676d7ba43e27a3370e5684d3ada61a0 2021-06-23T07:05:35.912 INFO:scrape:3 jobs: ['6147988', '6147996', '6147992'] 2021-06-23T07:05:35.913 INFO:scrape:suites intersection: ['1-ceph-install/mimic', '2-workload/{blogbench', '4-octopus', '5-msgr2', '6-final-workload/{blogbench', 'ec-rados-default', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_latest}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] 2021-06-23T07:05:35.914 INFO:scrape:suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-msgr2', '6-final-workload/{blogbench', 'ec-rados-default', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_latest}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] 2021-06-23T07:05:35.915 INFO:scrape: 2021-06-23T07:05:35.916 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/ failed: 3 dead: 13 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6147988] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 00:06:04 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147988/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=761c410007f746fab05f381c4712e839 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1 =aa3f3e3ca676d7ba43e27a3370e5684d3ada61a0 [6147992] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 00:06:03 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147992/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=093c9f140d5c445db75337ac8779785b Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1 =aa3f3e3ca676d7ba43e27a3370e5684d3ada61a0 [6147996] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 00:06:01 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147996/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd194b3860854097b952afbc2b92c2e4 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1 =aa3f3e3ca676d7ba43e27a3370e5684d3ada61a0 Dead ================================================================= [6147985] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147985/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147985/ Error reimaging machines: Failed to power on gibba039 [6147986] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147986/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147986/ Error reimaging machines: Failed to power on gibba039 [6147987] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147987/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147987/ Error reimaging machines: Failed to power on gibba034 [6147989] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147989/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147989/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [6147990] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147990/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147990/ Error reimaging machines: Failed to power on gibba034 [6147991] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147991/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147991/ Error reimaging machines: Failed to power on gibba034 [6147993] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147993/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147993/ Error reimaging machines: Failed to power on gibba039 [6147994] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147994/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147994/ Error reimaging machines: Failed to power on gibba034 [6147995] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147995/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147995/ Error reimaging machines: Failed to power on gibba034 [6147997] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147997/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147997/ Error reimaging machines: Failed to power on gibba034 [6147998] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147998/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147998/ Error reimaging machines: Failed to power on gibba034 [6147999] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147999/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6147999/ Error reimaging machines: Failed to power on gibba034 [6148000] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6148000/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-03_02:30:03-upgrade:mimic-x-octopus-distro-basic-gibba/6148000/ Error reimaging machines: Failed to power on gibba039