2021-01-14T17:20:36.430 INFO:root:teuthology version: 1.0.0-7ba49335 2021-01-14T17:20:36.432 INFO:teuthology.results:seed: '8723' 2021-01-14T20:00:40.856 INFO:root:teuthology version: 1.0.0-7ba49335 2021-01-14T20:00:40.895 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-15T07:49:53.270 INFO:teuthology.results:Tests finished! gathering results... 2021-01-15T07:49:58.282 INFO:scrape:Found 18 jobs 2021-01-15T07:49:58.476 INFO:scrape:Found 2 distinct failure reasons 2021-01-15T07:49:58.477 INFO:scrape:Dead: b'2021-01-15T07:47:25.224 DEBUG:teuthology.exit:Finished running handlers' 2021-01-15T07:49:58.477 INFO:scrape:5 jobs: ['5786638', '5786623', '5786633', '5786627', '5786629'] 2021-01-15T07:49:58.477 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-pacific', 'start}', 'thrashosds-health', 'ubuntu_18.04}'] 2021-01-15T07:49:58.478 INFO:scrape:suites union: ['1-ceph-install/pacific', '1-nautilus-install/pacific', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-01-15T07:49:58.479 INFO:scrape: 2021-01-15T07:49:58.479 INFO:scrape:Failure: Command failed on smithi080 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/yuriw/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 770369c04e5ab5930767a4db10fefcc3bf0b3894' 2021-01-15T07:49:58.479 INFO:scrape:4 jobs: ['5786630', '5786624', '5786637', '5786628'] 2021-01-15T07:49:58.480 INFO:scrape:suites intersection: ['1-ceph-install/pacific', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-01-15T07:49:58.480 INFO:scrape:suites union: ['1-ceph-install/pacific', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-01-15T07:49:58.481 INFO:scrape: 2021-01-15T07:49:58.483 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/ failed: 4 dead: 5 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [5786624] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 03:56:55 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786624/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b671e723eec4eeca41b05c5c8eec5fb Command failed on smithi120 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/yuriw/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 770369c04e5ab5930767a4db10fefcc3bf0b3894' [5786628] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 04:00:26 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786628/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fed2032a86d4bec9fd2a8a9964ac461 Command failed on smithi093 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/yuriw/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 770369c04e5ab5930767a4db10fefcc3bf0b3894' [5786630] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 02:43:41 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786630/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786630/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fad8e2ca3194f70a9cf4114d8bf4718 Command failed on smithi080 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/yuriw/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 770369c04e5ab5930767a4db10fefcc3bf0b3894' [5786637] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 04:33:15 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786637/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786637/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87c1d72c034b4da0a37ca07fefed2852 Command failed on smithi193 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/yuriw/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 770369c04e5ab5930767a4db10fefcc3bf0b3894' Dead ================================================================= [5786623] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786623/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786623/ [5786627] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786627/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786627/ [5786629] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786629/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786629/ [5786633] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786633/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786633/ [5786638] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786638/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786638/ Pass ================================================================= [5786622] upgrade:octopus-x/parallel/{0-start 1-tasks distro1$/{ubuntu_18.04_podman} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:42 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786622/ [5786625] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 02:02:31 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786625/ [5786626] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:10:01 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786626/ [5786631] upgrade:octopus-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:46 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786631/ [5786632] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 03:42:30 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786632/ [5786634] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 02:25:59 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786634/ [5786635] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:03:11 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786635/ [5786636] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/pacific 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 02:01:36 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786636/ [5786639] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:02:07 info: http://pulpito.ceph.com/teuthology-2021-01-14_17:07:59-upgrade:octopus-x-master-distro-basic-smithi/5786639/