2021-09-10T18:43:49.099 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-10T18:43:49.101 INFO:teuthology.results:subset: '1/3' 2021-09-10T18:43:49.101 INFO:teuthology.results:seed: '831' 2021-09-10T19:56:32.837 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-10T19:56:32.872 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-10T21:06:37.992 INFO:teuthology.results:Tests finished! gathering results... 2021-09-10T21:06:49.682 INFO:scrape:Found 16 jobs 2021-09-10T21:06:49.684 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384282/teuthology.log 2021-09-10T21:07:04.064 INFO:scrape:Found 2 distinct failure reasons 2021-09-10T21:07:04.065 INFO:scrape:Dead: None 2021-09-10T21:07:04.066 INFO:scrape:5 jobs: ['6384282', '6384285', '6384281', '6384288', '6384290'] 2021-09-10T21:07:04.066 INFO:scrape: 2021-09-10T21:07:04.067 INFO:scrape:Failure: Command failed (s3 tests against rgw) on gibba001 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2021-09-10T21:07:04.068 INFO:scrape:11 jobs: ['6384284', '6384291', '6384278', '6384289', '6384292', '6384286', '6384280', '6384283', '6384277', '6384279', '6384287'] 2021-09-10T21:07:04.068 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-10T21:07:04.069 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-10T21:07:04.070 INFO:scrape: 2021-09-10T21:07:04.071 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/ failed: 11 dead: 5 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6384277] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:23:25 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384277/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c58d1cf6f4d24efe8987805c4cab14bd Command failed (s3 tests against rgw) on gibba005 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384278] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:09:41 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384278/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384278/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d4bb5f597d44e02bb427c6445abad0c Command failed (s3 tests against rgw) on gibba032 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384279] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:25:13 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384279/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2817c3cc4f446d1abcac1e378488855 Command failed (s3 tests against rgw) on gibba011 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384280] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:02:48 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384280/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36aeece7c5894b5e8e887b78d4e72367 Command failed (s3 tests against rgw) on gibba004 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384283] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:23:29 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384283/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34b2ee82a10647ada46621478f808f92 Command failed (s3 tests against rgw) on gibba022 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384284] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:03:24 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384284/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47caa502bfe34cd5b43b6861b0cfdfa9 Command failed (s3 tests against rgw) on gibba001 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384286] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:05:34 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384286/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384286/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9016c68e511c480ca5d8492d92069c76 Command failed (s3 tests against rgw) on gibba005 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384287] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:24:37 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384287/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19dbbd867229482cbf94506120c78b58 Command failed (s3 tests against rgw) on gibba011 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384289] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:24:49 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384289/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384289/ sentry: https://sentry.ceph.com/organizations/ceph/?query=035cda481d53479ab0d62af606c8a088 Command failed (s3 tests against rgw) on gibba022 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384291] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:26:19 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384291/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1f64d763cd14fc992560b557ce347f3 Command failed (s3 tests against rgw) on gibba011 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6384292] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:00:53 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384292/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384292/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b97ee645f314a2586cae13db1b1bcbe Command failed (s3 tests against rgw) on gibba004 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" Dead ================================================================= [6384281] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384281/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384281/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384282] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384282/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384282/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384285] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384285/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384285/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384288] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384288/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384288/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384290] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384290/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-10_18:25:39-rgw-wip-fix-bi-test-error-distro-basic-gibba/6384290/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds