2021-07-03T14:23:07.686 INFO:root:teuthology version: 1.1.0-26631ede 2021-07-03T14:23:07.688 INFO:teuthology.results:seed: '7579' 2021-07-03T15:40:59.889 INFO:root:teuthology version: 1.1.0-26631ede 2021-07-03T15:40:59.918 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-07-03T18:27:11.182 INFO:teuthology.results:Tests finished! gathering results... 2021-07-03T18:27:14.440 INFO:scrape:Found 32 jobs 2021-07-03T18:27:16.790 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251196/teuthology.log 2021-07-03T18:27:21.793 INFO:scrape:Found 2 distinct failure reasons 2021-07-03T18:27:21.794 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' 2021-07-03T18:27:21.795 INFO:scrape:12 jobs: ['6251183', '6251187', '6251207', '6251195', '6251199', '6251200', '6251180', '6251203', '6251191', '6251192', '6251184', '6251204'] 2021-07-03T18:27:21.796 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', '2-workload/{blogbench', '4-octopus', '5-final-workload/{blogbench', 'ec-rados-default', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2021-07-03T18:27:21.796 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-final-workload/{blogbench', 'ec-rados-default', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2021-07-03T18:27:21.797 INFO:scrape: 2021-07-03T18:27:21.797 INFO:scrape:Dead: None 2021-07-03T18:27:21.797 INFO:scrape:6 jobs: ['6251196', '6251211', '6251208', '6251202', '6251197', '6251188'] 2021-07-03T18:27:21.798 INFO:scrape: 2021-07-03T18:27:21.801 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/ failed: 12 dead: 6 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [6251180] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:45:19 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=293117840373445f9fdc3f1579592751 Command failed (workunit test cls/test_cls_rgw.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251183] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:42:41 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251183/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e78af18c417413cbcb701d0f465360d Command failed (workunit test cls/test_cls_rgw.sh) on smithi096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251184] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:44:38 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251184/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1f9ba3f6d5747be8a59a1c20fc0d3fb Command failed (workunit test cls/test_cls_rgw.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251187] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:42:24 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251187/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c5f633de6994e3d8800cbaf22d98fa6 Command failed (workunit test cls/test_cls_rgw.sh) on smithi205 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251191] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:42:06 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251191/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a403884e3d3a4eadb9f12013b23e93db Command failed (workunit test cls/test_cls_rgw.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251192] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:45:50 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251192/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf37937cd3af4cf5a494606a9525d78a Command failed (workunit test cls/test_cls_rgw.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251195] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:42:03 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251195/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=546fb21d20174a34a3f839adb9cdfb23 Command failed (workunit test cls/test_cls_rgw.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251199] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:38:47 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251199/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9848b74d302747158d27b9d34a8a2102 Command failed (workunit test cls/test_cls_rgw.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251200] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:41:46 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251200/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=489df1dc931b40a1aefd9c019182258f Command failed (workunit test cls/test_cls_rgw.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251203] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:40:15 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251203/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2199bd926863479b8e4d9497fa64205f Command failed (workunit test cls/test_cls_rgw.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251204] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:42:48 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251204/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c116597bc15247c79b3baf1956a227ba Command failed (workunit test cls/test_cls_rgw.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6251207] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:39:54 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251207/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7b70cf098264581a643927801eaf96c Command failed (workunit test cls/test_cls_rgw.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' Dead ================================================================= [6251188] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251188/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251188/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6251196] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251196/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251196/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6251197] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251197/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251197/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6251202] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251202/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251202/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6251208] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251208/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251208/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6251211] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251211/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251211/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds Pass ================================================================= [6251181] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 02:42:04 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251181/ [6251182] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 01:51:45 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251182/ [6251185] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:42:35 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251185/ [6251186] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 01:24:16 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251186/ [6251189] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 02:45:42 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251189/ [6251190] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 01:32:59 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251190/ [6251193] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:46:04 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251193/ [6251194] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 01:42:21 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251194/ [6251198] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 01:32:43 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251198/ [6251201] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:56:42 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251201/ [6251205] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 02:52:31 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251205/ [6251206] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 01:18:24 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251206/ [6251209] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:46:05 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251209/ [6251210] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 01:51:06 info: http://pulpito.ceph.com/teuthology-2021-07-03_14:23:01-upgrade:nautilus-x-pacific-distro-basic-smithi/6251210/