2021-08-30T14:02:55.424 INFO:root:teuthology version: 1.1.0-df69d6db
2021-08-30T14:02:55.425 INFO:teuthology.results:seed: '6783'
2021-08-30T14:35:29.457 INFO:root:teuthology version: 1.1.0-df69d6db
2021-08-30T14:35:29.485 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-08-30T17:15:41.224 INFO:teuthology.results:Tests finished! gathering results...
2021-08-30T17:15:52.050 INFO:scrape:Found 72 jobs
2021-08-30T17:15:52.257 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366741/teuthology.log matches
2021-08-30T17:15:52.258 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:52.294 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366741/teuthology.log matches
2021-08-30T17:15:52.295 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:52.326 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366741/teuthology.log matches
2021-08-30T17:15:52.327 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:54.210 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366770/teuthology.log matches
2021-08-30T17:15:54.211 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:54.493 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366752/teuthology.log matches
2021-08-30T17:15:54.495 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:57.035 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366746/teuthology.log matches
2021-08-30T17:15:57.037 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:57.194 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366780/teuthology.log matches
2021-08-30T17:15:57.195 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:15:59.790 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366804/teuthology.log matches
2021-08-30T17:15:59.791 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:00.240 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366748/teuthology.log matches
2021-08-30T17:16:00.242 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:01.942 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366772/teuthology.log matches
2021-08-30T17:16:01.955 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:02.507 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366756/teuthology.log matches
2021-08-30T17:16:02.508 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:02.846 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366794/teuthology.log matches
2021-08-30T17:16:02.849 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:03.028 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366734/teuthology.log matches
2021-08-30T17:16:03.035 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:03.194 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366754/teuthology.log matches
2021-08-30T17:16:03.195 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:05.969 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366747/teuthology.log matches
2021-08-30T17:16:05.970 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:07.196 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366738/teuthology.log matches
2021-08-30T17:16:07.197 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:08.676 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366773/teuthology.log matches
2021-08-30T17:16:08.677 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:08.846 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366733/teuthology.log matches
2021-08-30T17:16:08.847 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:10.356 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366757/teuthology.log matches
2021-08-30T17:16:10.357 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:10.586 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366761/teuthology.log matches
2021-08-30T17:16:10.587 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:10.746 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366788/teuthology.log matches
2021-08-30T17:16:10.747 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:11.240 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366781/teuthology.log matches
2021-08-30T17:16:11.241 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:11.481 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366753/teuthology.log matches
2021-08-30T17:16:11.483 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:11.853 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366777/teuthology.log matches
2021-08-30T17:16:11.854 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:12.047 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366737/teuthology.log matches
2021-08-30T17:16:12.048 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:12.214 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366751/teuthology.log matches
2021-08-30T17:16:12.215 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:12.412 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366758/teuthology.log matches
2021-08-30T17:16:12.413 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:14.263 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366739/teuthology.log matches
2021-08-30T17:16:14.264 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:16.163 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366782/teuthology.log matches
2021-08-30T17:16:16.164 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:17.173 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366785/teuthology.log matches
2021-08-30T17:16:17.174 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:17.331 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366736/teuthology.log matches
2021-08-30T17:16:17.332 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:17.820 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366791/teuthology.log matches
2021-08-30T17:16:17.822 WARNING:scrape:Not-understood crash indication 
2021-08-30T17:16:18.172 INFO:scrape:Found 6 distinct failure reasons
2021-08-30T17:16:18.173 INFO:scrape:Failure: Command failed on smithi078 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest'
2021-08-30T17:16:18.174 INFO:scrape:30 jobs
2021-08-30T17:16:18.175 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls']
2021-08-30T17:16:18.175 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'datacache/rgw-ioc-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-30T17:16:18.176 INFO:scrape:
2021-08-30T17:16:18.176 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2021-08-30T17:16:18.176 INFO:scrape:5 jobs: ['6366792', '6366766', '6366793', '6366784', '6366742']
2021-08-30T17:16:18.177 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls']
2021-08-30T17:16:18.177 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-30T17:16:18.178 INFO:scrape:
2021-08-30T17:16:18.178 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'
2021-08-30T17:16:18.179 INFO:scrape:33 jobs
2021-08-30T17:16:18.179 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls']
2021-08-30T17:16:18.180 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'datacache/rgw-ioc-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-30T17:16:18.180 INFO:scrape:
2021-08-30T17:16:18.181 INFO:scrape:Crash: Command failed on smithi060 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest'
2021-08-30T17:16:18.181 INFO:scrape:ceph version 17.0.0-7367-g0ebb54dc (0ebb54dc8a416e7ee409452e3283bbcbaafb9121) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0x7fbc953dedc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fbc9769fcbc]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x284e7f) [0x7fbc9769fe7f]
 6: (Monitor::~Monitor()+0xb35) [0x55a5daafd6e5]
 7: (Monitor::~Monitor()+0xd) [0x55a5daafd73d]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-08-30T17:16:18.182 INFO:scrape:1 jobs: ['6366768']
2021-08-30T17:16:18.183 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/valgrind}']
2021-08-30T17:16:18.183 INFO:scrape:
2021-08-30T17:16:18.183 INFO:scrape:Failure: "2021-08-30T14:24:49.856134+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2021-08-30T17:16:18.184 INFO:scrape:2 jobs: ['6366771', '6366740']
2021-08-30T17:16:18.184 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-ioc-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls']
2021-08-30T17:16:18.185 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-ioc-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-30T17:16:18.185 INFO:scrape:
2021-08-30T17:16:18.186 INFO:scrape:Failure: Command failed (workunit test rgw/run-reshard.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh'
2021-08-30T17:16:18.186 INFO:scrape:1 jobs: ['6366745']
2021-08-30T17:16:18.187 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-ioc-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}']
2021-08-30T17:16:18.187 INFO:scrape:
2021-08-30T17:16:18.191 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/
failed:  72
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6366733]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:35:36
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366733/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366733/
sentry: https://sentry.ceph.com/organizations/ceph/?query=500dbd0e02ce49aca67c2a031804e774

    Command failed on smithi137 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366734]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:47:40
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366734/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366734/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d3903a0a671485ab3f7b2760552d5fb

    Command failed on smithi122 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366735]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:30:11
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366735/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366735/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a97d4d070e3541b59b38c8132c3a40f3

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi012 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366736]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:40:31
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366736/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366736/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4e884aaca314d91ae95d0463cbd7f1b

    Command failed on smithi093 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366737]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:27:12
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366737/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366737/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d97aa8e4c47f4dc0b3e79d7ecd43deda

    Command failed on smithi027 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366738]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:47:50
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366738/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366738/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d32d56f78ec646cbb2aa0d17ac006596

    Command failed on smithi114 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366739]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:31:34
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366739/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366739/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8fc542ab8fe74ebe894f3b9f33842e89

    Command failed on smithi134 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366740]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:27:55
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366740/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366740/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c89b9db14aa0400dbb1457c2b1f05d68

    "2021-08-30T14:32:03.882107+0000 mon.a (mon.0) 288 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6366741]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:27:06
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366741/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366741/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9981ce20e23f4662ad652023ec99e4d2

    Command failed on smithi078 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366742]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:27:14
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366742/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366742/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aa3459d5a01e404bb3f16ca33b603e49

    reached maximum tries (90) after waiting for 540 seconds

[6366743]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:17:43
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366743/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366743/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3222c792e5da420d9112167a43ff07e9

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi050 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366744]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:29:49
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366744/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366744/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b2d61947420408d9e1ea618fca03619

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi058 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366745]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:25:36
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366745/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cff6b509ec6c45868df6998cc478387c

    Command failed (workunit test rgw/run-reshard.sh) on smithi042 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh'

[6366746]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:40:43
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366746/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366746/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd65889e0ef641ddbd5a57d9dfd701b6

    Command failed on smithi159 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366747]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:38:29
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366747/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0fa40b708edd4c1bae5723d07722537c

    Command failed on smithi115 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366748]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:47:41
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366748/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366748/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd02b7e3a98d4e9e812e95296c6025fa

    Command failed on smithi089 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366749]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:17:58
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366749/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366749/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eddba25dd7534450a76da7cb1eb7ded0

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi063 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366750]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:26:42
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366750/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366750/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e38896230e8e4e4da878a5b0f08e6a59

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi120 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366751]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:34:11
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366751/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366751/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35f320c565b34ccab5e0d1ce4bd4d324

    Command failed on smithi049 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366752]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:44:50
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366752/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2357f29626f242d39dbcf9c61fb72849

    Command failed on smithi077 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366753]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:37:27
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366753/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366753/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11e0ae5d594a4ccf88e264341b963633

    Command failed on smithi018 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366754]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:45:57
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366754/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366754/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be9eb86b7bf44487b0eb6ebb6d8496aa

    Command failed on smithi178 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366755]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:27:55
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366755/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366755/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff8dcee2e1844b24acd7aaf7763f3606

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi019 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366756]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:45:53
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366756/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366756/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d721220f13b44b4c8921b58900b8a90d

    Command failed on smithi064 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366757]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:30:12
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366757/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366757/
sentry: https://sentry.ceph.com/organizations/ceph/?query=69eefa09f4584c4092badd13bf766cfb

    Command failed on smithi117 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366758]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:39:40
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366758/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366758/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a13bdceccc564577a9cac0d7d1657a2d

    Command failed on smithi039 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366759]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:23:16
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366759/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366759/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36b8daf9ba2345ecb3392972b18d8961

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi109 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366760]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:28:00
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366760/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366760/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b51cf26f8dd445c292b00023596abeba

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi152 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366761]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:38:05
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366761/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366761/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8be14cb193b04a008aa58228faebd6f0

    Command failed on smithi143 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366762]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:30:57
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366762/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366762/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c6aed9759e8416895267f627291f8fc

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi025 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366763]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:28:46
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366763/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366763/
sentry: https://sentry.ceph.com/organizations/ceph/?query=efb245786c5e4e55a47f68458578f706

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi038 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366764]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:26:23
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366764/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366764/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8919ce033832414d85b752306f46635f

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi053 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366765]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:30:40
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366765/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366765/
sentry: https://sentry.ceph.com/organizations/ceph/?query=627dd9788dff4379864b80dbb9d1e28a

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi071 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366766]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:25:18
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366766/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366766/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7f5f486f4a034c12b776c30d58bb4c38

    reached maximum tries (90) after waiting for 540 seconds

[6366767]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:17:40
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366767/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366767/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54760a1d62ce4240b03afdc2749540bb

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi100 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366768]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:39:02
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366768/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366768/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6dd190798a694432921485217180d113

    Command failed on smithi060 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366769]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:25:51
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366769/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366769/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d80d5a3422a4ff18a9b0990182b68ef

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi017 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366770]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:51:38
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366770/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366770/
sentry: https://sentry.ceph.com/organizations/ceph/?query=125cc381812a42c48a4b5a2eeffc2af2

    Command failed on smithi016 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366771]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:20:06
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366771/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366771/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1708e42bbb9c438295af3b8d878318ff

    "2021-08-30T14:24:49.856134+0000 mon.a (mon.0) 132 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6366772]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:49:54
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366772/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366772/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51bb6ad47b6844a6909bf5325e7e90bd

    Command failed on smithi057 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366773]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:36:10
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366773/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366773/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8aefba7e4770419786d2047e777db2b1

    Command failed on smithi116 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366774]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:30:13
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366774/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366774/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8354d51f875b4f0a915115c47d3c6f40

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi156 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366775]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:18:01
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366775/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366775/
sentry: https://sentry.ceph.com/organizations/ceph/?query=84df7e5a958342dab630d479cb8d7cac

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi062 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366776]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:31:21
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366776/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366776/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f8007b49bf642df955b9c59e5355dd0

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi065 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366777]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:25:31
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366777/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366777/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d696eae533d43e2b9535b1062606519

    Command failed on smithi013 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366778]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:28:52
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366778/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366778/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ecf06dbca3c643f19d2cb2736e2f077f

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi151 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366779]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:27:31
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366779/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366779/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e81125aeb8ce43bca5799b92c0dd69db

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi132 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366780]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:50:30
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366780/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366780/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5cc9338a2ae4ed3941302feed1b3e25

    Command failed on smithi037 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366781]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:28:52
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366781/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366781/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6d1754a0c1fd4d8db87002fa05e0d990

    Command failed on smithi043 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366782]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:47:36
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366782/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366782/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8ebda4356cf54f9aace69771dcfe555d

    Command failed on smithi091 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366783]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:30:42
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366783/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366783/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f081bf35de014a94ae66cd5778aba2b6

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi166 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366784]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:26:47
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366784/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366784/
sentry: https://sentry.ceph.com/organizations/ceph/?query=151512f79c52464f830d717506cb7fe4

    reached maximum tries (90) after waiting for 540 seconds

[6366785]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:37:35
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366785/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366785/
sentry: https://sentry.ceph.com/organizations/ceph/?query=134ac06e91a84841a214226a0357639f

    Command failed on smithi103 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366786]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:27:45
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366786/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366786/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d69259f84b8743f7bcefec9631180a40

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi059 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366787]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:43
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366787/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366787/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e2a1aee9fb84d67a9057c284d88cbff

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi079 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366788]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:46:12
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366788/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366788/
sentry: https://sentry.ceph.com/organizations/ceph/?query=686769b2df5a45da84dadb70b7e7f98e

    Command failed on smithi045 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366789]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:30:03
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366789/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33968fe1be3b44cda53309af624dcd0e

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi150 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366790]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:29:50
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366790/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe21fa02b8464da5a6bf02728e4bc58e

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi101 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366791]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   02:36:17
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366791/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ac26208d08644ae8979cbf9b1af67fb5

    Command failed on smithi022 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366792]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:25:15
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366792/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366792/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2695e6b507a4d21a08769259024b747

    reached maximum tries (90) after waiting for 540 seconds

[6366793]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:26:55
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366793/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8549eade62db4c5aad1e338a3ce36201

    reached maximum tries (90) after waiting for 540 seconds

[6366794]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:47:07
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366794/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366794/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b03563260b1b40fdb9947b29d7d3aa82

    Command failed on smithi028 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'

[6366795]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:26:14
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366795/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78ecbb4771e140c59171397e0c6c52a7

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi107 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366796]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:25:03
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366796/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366796/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4169e623a03944af82447865aa4fcc6d

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi139 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366797]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:28:47
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366797/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a76d01389a3c4665a720a2e3a08d5fea

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi023 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366798]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:19:09
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366798/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366798/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f13ee3e4b274c8ead3dd2847f9812ca

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi092 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366799]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:14:54
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366799/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=74003bde9f524d8e9780dd1724a74717

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi051 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366800]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:17:40
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366800/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=858085bb5d8f4e6990c23e3feb9c7df6

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi079 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366801]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:14
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366801/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=319c8acd417f48bcafb513d8751c9eeb

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi063 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366802]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:16:36
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366802/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366802/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4cf14f5e581f4312bcde1e2f86ba61f1

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi050 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366803]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-ioc-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:02
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366803/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a924ab6924943cdae51f980c2b6ce5c

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi131 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ebb54dc8a416e7ee409452e3283bbcbaafb9121
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6366804]  rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   02:30:04
info:   http://pulpito.ceph.com/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366804/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2021-08-30_14:02:38-rgw:verify-wip-ioc_cache_i02-distro-basic-smithi/6366804/
sentry: https://sentry.ceph.com/organizations/ceph/?query=15050684d5c54ce1889cf5f42ef5710c

    Command failed on smithi062 with status 1: 'cd
    /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean
    test --rerun-tasks --no-build-cache --tests ObjectTest'