2021-01-12T02:31:10.911 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-12T02:31:10.913 INFO:teuthology.results:seed: '1301' 2021-01-12T02:38:45.002 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-12T02:38:45.028 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-12T14:38:57.338 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-01-12T14:39:01.260 INFO:scrape:Found 16 jobs 2021-01-12T14:39:08.206 INFO:scrape:Found 2 distinct failure reasons 2021-01-12T14:39:08.207 INFO:scrape:Failure: Command failed on smithi124 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2021-01-12T14:39:08.207 INFO:scrape:8 jobs: ['5778696', '5778694', '5778672', '5778702', '5778688', '5778678', '5778686', '5778680'] 2021-01-12T14:39:08.208 INFO:scrape:suites intersection: ['1-ceph-install/mimic', '2-workload/{blogbench', '4-octopus', '5-msgr2', '6-final-workload/{blogbench', 'ec-rados-default', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_latest}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] 2021-01-12T14:39:08.208 INFO:scrape:suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-msgr2', '6-final-workload/{blogbench', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_latest}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] 2021-01-12T14:39:08.209 INFO:scrape: 2021-01-12T14:39:08.209 INFO:scrape:Dead: b'2021-01-12T14:31:21.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 0)...' 2021-01-12T14:39:08.210 INFO:scrape:3 jobs: ['5778674', '5778700', '5778690'] 2021-01-12T14:39:08.210 INFO:scrape: 2021-01-12T14:39:08.212 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/ failed: 8 dead: 1 running: 0 waiting: 2 queued: 0 passed: 5 Fail ================================================================= [5778672] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 01:13:53 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778672/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bbce0268d7804f60aba4f7ddfce0f2cf Command failed on smithi019 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778678] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 01:09:27 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778678/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778678/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e9bd81f74be4d9e996e396bbe670024 Command failed on smithi006 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778680] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 01:12:59 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778680/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778680/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fa80114207b422397fc438f7047b7f3 Command failed on smithi181 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778686] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 01:10:03 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778686/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778686/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c26788cac962493a95891bf82bf276a2 Command failed on smithi033 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778688] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 01:12:56 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778688/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778688/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50df0071b7fd491bb061521e24cb82d1 Command failed on smithi047 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778694] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 01:09:34 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778694/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=250df1d344314adaa56d7079967377e6 Command failed on smithi028 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778696] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 01:12:42 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778696/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d57b28a3fded441aaa7e112229ad6b99 Command failed on smithi124 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5778702] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 01:09:01 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778702/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf18b15f382440cab62c231e3a3c1720 Command failed on smithi014 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' Dead ================================================================= [5778674] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778674/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778674/ Waiting ================================================================= [5778690] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778690/ [5778700] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778700/ Pass ================================================================= [5778676] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 00:51:06 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778676/ [5778682] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:09:12 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778682/ [5778684] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 00:48:27 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778684/ [5778692] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 00:50:51 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778692/ [5778698] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:14:29 info: http://pulpito.ceph.com/teuthology-2021-01-12_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5778698/