2021-09-03T19:21:58.205 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-03T19:21:58.207 INFO:teuthology.results:subset: '1/3' 2021-09-03T19:21:58.207 INFO:teuthology.results:seed: '831' 2021-09-03T19:22:54.335 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-03T19:22:54.336 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-03T19:58:56.828 INFO:teuthology.results:Tests finished! gathering results... 2021-09-03T19:58:57.348 INFO:scrape:Found 16 jobs 2021-09-03T19:58:58.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374210/teuthology.log 2021-09-03T19:58:58.846 INFO:scrape:Found 2 distinct failure reasons 2021-09-03T19:58:58.847 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on smithi028 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" 2021-09-03T19:58:58.848 INFO:scrape:15 jobs: ['6374220', '6374213', '6374218', '6374217', '6374216', '6374221', '6374212', '6374223', '6374222', '6374209', '6374214', '6374208', '6374211', '6374219', '6374215'] 2021-09-03T19:58:58.848 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-03T19:58:58.849 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-03T19:58:58.849 INFO:scrape: 2021-09-03T19:58:58.850 INFO:scrape:Dead: None 2021-09-03T19:58:58.850 INFO:scrape:1 jobs: ['6374210'] 2021-09-03T19:58:58.850 INFO:scrape: 2021-09-03T19:58:58.851 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/ failed: 15 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6374208] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:21:32 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374208/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0d93f6322a645f780edef8412a87d6d Command failed (ragweed tests against rgw) on smithi079 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374209] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:22:03 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374209/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0670a22e09b4bdb85c08e175e2cfe0f Command failed (ragweed tests against rgw) on smithi077 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374211] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:23:00 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374211/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e907a48afb44a4c9572e77a95a58fae Command failed (ragweed tests against rgw) on smithi052 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374212] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:18:46 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374212/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374212/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8696a1c86874f23814f551a32cf9a06 Command failed (ragweed tests against rgw) on smithi016 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374213] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:23:30 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374213/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52d5fe91d6bd4cc6a496bde40ea505a6 Command failed (ragweed tests against rgw) on smithi050 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374214] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:21:28 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374214/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a85e773c3880420ab7af75d1d5350453 Command failed (ragweed tests against rgw) on smithi153 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374215] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:23:18 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374215/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c3bf8bd0f0e41a6b08d5e635f6d26dd Command failed (ragweed tests against rgw) on smithi068 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374216] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:23:08 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374216/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=541fe6d8ca6a4250931426d822262346 Command failed (ragweed tests against rgw) on smithi107 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374217] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:44 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374217/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc47529482974be78464bfee0b391457 Command failed (ragweed tests against rgw) on smithi132 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374218] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:21:23 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374218/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdb4e2a1bd6e454080618ac9ead09325 Command failed (ragweed tests against rgw) on smithi099 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374219] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:09 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374219/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff21de3f385b41e69a347819f2d384d4 Command failed (ragweed tests against rgw) on smithi063 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374220] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:22:42 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374220/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3a1a19ff1a5412da9c9a5dc529bd1fa Command failed (ragweed tests against rgw) on smithi028 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374221] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:26 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374221/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374221/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78d6296a0b6b4fa3b4fd4898e70f769b Command failed (ragweed tests against rgw) on smithi057 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374222] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:22:39 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374222/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43bb3b28b3a445d49555e0f48b164da8 Command failed (ragweed tests against rgw) on smithi154 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6374223] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:20:55 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374223/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbb04c65e2af4b8d93432575853f9708 Command failed (ragweed tests against rgw) on smithi027 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" Dead ================================================================= [6374210] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374210/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_19:21:52-rgw-wip-eric-testing-2-distro-basic-smithi/6374210/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds