2021-01-22T14:23:45.245 INFO:root:teuthology version: 1.0.0-fe79ba1b 2021-01-22T14:23:45.246 INFO:teuthology.results:seed: '225' 2021-01-22T14:32:17.918 INFO:root:teuthology version: 1.0.0-fe79ba1b 2021-01-22T14:32:17.942 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-22T18:34:22.157 INFO:teuthology.results:Tests finished! gathering results... 2021-01-22T18:34:24.254 INFO:scrape:Found 16 jobs 2021-01-22T18:34:30.627 INFO:scrape:Found 1 distinct failure reasons 2021-01-22T18:34:30.628 INFO:scrape:Failure: Command failed on smithi125 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2021-01-22T18:34:30.628 INFO:scrape:9 jobs: ['5815561', '5815566', '5815573', '5815569', '5815562', '5815570', '5815565', '5815564', '5815558'] 2021-01-22T18:34:30.629 INFO:scrape:suites intersection: ['start}', 'ubuntu_latest}'] 2021-01-22T18:34:30.629 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-octopus', '5-final-workload/{blogbench', '5-finish-upgrade', '6-octopus', '7-final-workload', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-ec-workload}', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2021-01-22T18:34:30.630 INFO:scrape: 2021-01-22T18:34:30.632 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [5815558] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 03:16:05 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815558/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815558/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1895298640c1490a9dbc8c4d1117b0c3 Command failed on smithi104 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815561] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 03:08:12 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815561/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b814c03b4b5a40369120bda833473d74 Command failed on smithi125 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815562] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 03:17:48 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815562/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815562/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df41f7d8fc954bd788cd32310ec7c9ff Command failed on smithi075 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815564] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:29:22 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815564/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815564/ Command failed on smithi091 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json' [5815565] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 03:07:55 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815565/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=214c29c488f348dca666e2e996aa08f8 Command failed on smithi160 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815566] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 03:16:39 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815566/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9daf60e9a2f46aa937761ecb22b6dde Command failed on smithi162 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815569] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 03:12:34 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815569/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99de2f48c135463c9d831a6f6e386a32 Command failed on smithi079 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815570] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} ----------------------------------------------------------------- time: 03:14:15 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815570/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f13c37580694a2a93e4ea58d1a48b61 Command failed on smithi191 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5815573] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 03:08:59 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815573/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815573/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a3103fef2944161bf280ea2fb864124 Command failed on smithi180 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' Pass ================================================================= [5815559] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:34:21 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815559/ [5815560] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:13:42 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815560/ [5815563] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:36:36 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815563/ [5815567] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:52:39 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815567/ [5815568] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:11:25 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815568/ [5815571] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:24:40 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815571/ [5815572] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:03:51 info: http://pulpito.ceph.com/teuthology-2021-01-22_14:23:03-upgrade:nautilus-x-octopus-distro-basic-smithi/5815572/