2021-09-16T09:08:32.438 INFO:root:teuthology version: 1.1.0-63133ca9 2021-09-16T09:08:32.439 INFO:teuthology.results:subset: '1/3' 2021-09-16T09:08:32.440 INFO:teuthology.results:seed: '831' 2021-09-16T09:36:28.958 INFO:root:teuthology version: 1.1.0-63133ca9 2021-09-16T09:36:28.961 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-16T10:31:33.045 INFO:teuthology.results:Tests finished! gathering results... 2021-09-16T10:31:34.087 INFO:scrape:Found 16 jobs 2021-09-16T10:31:37.966 INFO:scrape:Found 2 distinct failure reasons 2021-09-16T10:31:37.967 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on smithi179 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" 2021-09-16T10:31:37.967 INFO:scrape:15 jobs: ['6391867', '6391858', '6391860', '6391868', '6391859', '6391866', '6391869', '6391862', '6391864', '6391865', '6391871', '6391872', '6391861', '6391873', '6391863'] 2021-09-16T10:31:37.968 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-16T10:31:37.968 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-16T10:31:37.969 INFO:scrape: 2021-09-16T10:31:37.969 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2021-09-16T10:31:37.970 INFO:scrape:1 jobs: ['6391870'] 2021-09-16T10:31:37.970 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}'] 2021-09-16T10:31:37.971 INFO:scrape: 2021-09-16T10:31:37.972 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6391858] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:28:33 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391858/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4bc98a7e06e4ce6b9e5902368945c57 Command failed (ragweed tests against rgw) on smithi159 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391859] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:22:17 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391859/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47ec141a792f4bf6ae2b2394b1749b7b Command failed (ragweed tests against rgw) on smithi016 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391860] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:31:24 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391860/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83aa11e169de4c8ab53925c6185efd31 Command failed (ragweed tests against rgw) on smithi071 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391861] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:32 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391861/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f1d1d772677459f8e9aae3c3a7f563f Command failed (ragweed tests against rgw) on smithi125 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391862] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:29:27 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391862/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da8bfbd8c1e3435389917def0452c61c Command failed (ragweed tests against rgw) on smithi025 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391863] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:41 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391863/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9676f7a575ac49afa22588b200b141c2 Command failed (ragweed tests against rgw) on smithi082 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391864] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:20:04 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391864/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd958615253b48958bbd248104c7c7b7 Command failed (ragweed tests against rgw) on smithi065 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391865] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:22:28 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391865/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0abd622afd74e6daff12ee66a5dee51 Command failed (ragweed tests against rgw) on smithi031 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391866] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:28:07 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391866/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391866/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2bb8e46c1414636a6d481c9597509f7 Command failed (ragweed tests against rgw) on smithi041 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391867] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391867/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f94f6ab8259a4dc08c04dc7952b430cc Command failed (ragweed tests against rgw) on smithi179 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391868] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:31:07 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391868/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=288ecb204c504a66a9cd9f818da11018 Command failed (ragweed tests against rgw) on smithi168 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391869] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:02 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391869/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391869/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89358e92676342ff9b971e8133000924 Command failed (ragweed tests against rgw) on smithi023 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391870] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:24:09 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391870/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391870/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c5342a47dce4048a8d4aae092385b8f reached maximum tries (90) after waiting for 540 seconds [6391871] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:21 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391871/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fef84d757084113980dcb207319c052 Command failed (ragweed tests against rgw) on smithi029 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391872] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:28:29 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391872/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de3bd4b923804b9aba8c9304edde1d4e Command failed (ragweed tests against rgw) on smithi103 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6391873] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:10 info: http://pulpito.ceph.com/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391873/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-15_23:42:23-rgw-wip-eric-testing-2-distro-basic-smithi/6391873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8018fe0adb541d0aca7565a289b8e7a Command failed (ragweed tests against rgw) on smithi098 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'"