2021-09-15T00:00:21.009 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-15T00:00:21.011 INFO:teuthology.results:seed: '3860' 2021-09-15T15:01:11.595 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-15T15:01:11.625 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-15T16:32:18.417 INFO:teuthology.results:Tests finished! gathering results... 2021-09-15T16:32:19.895 INFO:scrape:Found 16 jobs 2021-09-15T16:32:20.421 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373853/teuthology.log 2021-09-15T16:32:20.423 INFO:scrape:Found 2 distinct failure reasons 2021-09-15T16:32:20.424 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=59c73c082189c0f279d86688707294694fd83e97 2021-09-15T16:32:20.424 INFO:scrape:1 jobs: ['6373865'] 2021-09-15T16:32:20.425 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '4-octopus', '5-final-workload/{blogbench', 'ec-rados-default', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2021-09-15T16:32:20.425 INFO:scrape: 2021-09-15T16:32:20.426 INFO:scrape:Dead: None 2021-09-15T16:32:20.426 INFO:scrape:4 jobs: ['6373853', '6373855', '6373852', '6373856'] 2021-09-15T16:32:20.427 INFO:scrape: 2021-09-15T16:32:20.428 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/ failed: 1 dead: 4 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [6373865] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 01:20:16 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373865/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aef0abb9515842dca6cae16ffb4138d4 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1 =59c73c082189c0f279d86688707294694fd83e97 Dead ================================================================= [6373852] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373852/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373852/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6373853] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373853/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373853/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6373855] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373855/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373855/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [6373856] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373856/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373856/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds Pass ================================================================= [6373850] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:08:27 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373850/ [6373851] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 03:30:18 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373851/ [6373854] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:06:23 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373854/ [6373857] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 01:59:33 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373857/ [6373858] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:05:44 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373858/ [6373859] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 04:09:57 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373859/ [6373860] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} time: 01:55:14 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373860/ [6373861] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 02:00:11 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373861/ [6373862] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:05:49 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373862/ [6373863] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:53:11 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373863/ [6373864] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} time: 02:00:14 info: http://pulpito.ceph.com/teuthology-2021-09-03_14:23:02-upgrade:nautilus-x-octopus-distro-basic-gibba/6373864/