2021-09-03T17:41:01.494 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-03T17:41:01.495 INFO:teuthology.results:subset: '1/3' 2021-09-03T17:41:01.496 INFO:teuthology.results:seed: '831' 2021-09-03T18:39:56.560 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-03T18:39:56.566 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-03T19:02:58.233 INFO:teuthology.results:Tests finished! gathering results... 2021-09-03T19:02:58.796 INFO:scrape:Found 16 jobs 2021-09-03T19:03:00.554 INFO:scrape:Found 2 distinct failure reasons 2021-09-03T19:03:00.555 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on gibba011 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" 2021-09-03T19:03:00.555 INFO:scrape:14 jobs: ['6373983', '6373972', '6373973', '6373976', '6373981', '6373979', '6373980', '6373971', '6373975', '6373978', '6373970', '6373974', '6373977', '6373982'] 2021-09-03T19:03:00.556 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-03T19:03:00.556 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-03T19:03:00.557 INFO:scrape: 2021-09-03T19:03:00.557 INFO:scrape:Dead: b'2021-09-03T19:01:25.540 DEBUG:teuthology.exit:Finished running handlers' 2021-09-03T19:03:00.558 INFO:scrape:2 jobs: ['6373984', '6373985'] 2021-09-03T19:03:00.558 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-03T19:03:00.559 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-03T19:03:00.559 INFO:scrape: 2021-09-03T19:03:00.560 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/ failed: 14 dead: 2 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6373970] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:19:28 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373970/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f2581d3a0924425861fd98ecc3a08cf Command failed (ragweed tests against rgw) on gibba005 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373971] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:16 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373971/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=410a28254e8b494c857227bc5eda5016 Command failed (ragweed tests against rgw) on gibba002 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373972] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373972/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e8dfef6bedc428197fe47ba9187ea16 Command failed (ragweed tests against rgw) on gibba001 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373973] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:18:12 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373973/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9d1ce9bb4574d2997ee7faaf61bd862 Command failed (ragweed tests against rgw) on gibba008 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373974] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:17:54 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373974/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373974/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7a8ba5f52d3499e812cf24fd1d5a320 Command failed (ragweed tests against rgw) on gibba004 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373975] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:18:23 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373975/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07dfceebc8c74873ab985aafbcab2ca2 Command failed (ragweed tests against rgw) on gibba011 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373976] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373976/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06f0e8177f19478fb2f7fcc8dbccb0ba Command failed (ragweed tests against rgw) on gibba027 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373977] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:18:47 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373977/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4ae22b341a34defbfded6601174c328 Command failed (ragweed tests against rgw) on gibba005 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373978] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:20:01 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373978/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373978/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31d8733a6ab446edad41a6b592feb146 Command failed (ragweed tests against rgw) on gibba001 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373979] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:17:48 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373979/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373979/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f26623159a01443da5ce4181dfbb48d2 Command failed (ragweed tests against rgw) on gibba002 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373980] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:19:23 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373980/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2be3f56bc8b649f980d91fe9c1a48887 Command failed (ragweed tests against rgw) on gibba008 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373981] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:10 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373981/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373981/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a07668c72b74470c82f9d949ffaa3ad1 Command failed (ragweed tests against rgw) on gibba004 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373982] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:26:58 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373982/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57e01f0ba287441598ac9cd6c9bcd92a Command failed (ragweed tests against rgw) on gibba027 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6373983] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:26:11 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373983/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8255d0e221ea4d4c91199b6f8aaf5f2a Command failed (ragweed tests against rgw) on gibba011 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" Dead ================================================================= [6373984] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373984/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373984/ [6373985] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373985/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-03_17:37:33-rgw-wip-eric-testing-1-distro-basic-gibba/6373985/