2021-09-08T03:59:08.346 INFO:root:teuthology version: 1.1.0-b135a4f8
2021-09-08T03:59:08.348 INFO:teuthology.results:subset: '1/3'
2021-09-08T03:59:08.348 INFO:teuthology.results:seed: '831'
2021-09-08T03:59:59.182 INFO:root:teuthology version: 1.1.0-b135a4f8
2021-09-08T03:59:59.188 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-09-08T05:34:05.286 INFO:teuthology.results:Tests finished! gathering results...
2021-09-08T05:34:05.788 INFO:scrape:Found 16 jobs
2021-09-08T05:34:07.013 INFO:scrape:Found 4 distinct failure reasons
2021-09-08T05:34:07.014 INFO:scrape:Failure: 
2021-09-08T05:34:07.015 INFO:scrape:7 jobs: ['6379633', '6379622', '6379635', '6379624', '6379628', '6379626', '6379631']
2021-09-08T05:34:07.015 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 's3tests-java', 's3tests}', 'tasks/{0-install']
2021-09-08T05:34:07.016 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/lockdep}', 'validater/valgrind}']
2021-09-08T05:34:07.016 INFO:scrape:
2021-09-08T05:34:07.017 INFO:scrape:Failure: "2021-09-08T04:19:23.194012+0000 mon.a (mon.0) 117 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2021-09-08T05:34:07.017 INFO:scrape:1 jobs: ['6379632']
2021-09-08T05:34:07.018 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/valgrind}']
2021-09-08T05:34:07.018 INFO:scrape:
2021-09-08T05:34:07.019 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e216c1ee110397eb48ef026c794021917db737f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'
2021-09-08T05:34:07.019 INFO:scrape:4 jobs: ['6379630', '6379625', '6379623', '6379627']
2021-09-08T05:34:07.020 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 's3tests-java', 's3tests}', 'tasks/{0-install']
2021-09-08T05:34:07.020 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/lockdep}', 'validater/valgrind}']
2021-09-08T05:34:07.021 INFO:scrape:
2021-09-08T05:34:07.021 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2021-09-08T05:34:07.022 INFO:scrape:1 jobs: ['6379629']
2021-09-08T05:34:07.022 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'tasks/{0-install', 'validater/lockdep}']
2021-09-08T05:34:07.023 INFO:scrape:
2021-09-08T05:34:07.024 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/
failed:  13
dead:    0
running: 0
waiting: 0
queued:  0
passed:  3


Fail
=================================================================
[6379622]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:16:21
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379622/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379622/
sentry: https://sentry.ceph.com/organizations/ceph/?query=040cbfa3e73747649cf086ee3c27ac46

[6379623]  rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:14:11
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379623/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379623/
sentry: https://sentry.ceph.com/organizations/ceph/?query=52ca8ec3908149b0884409cf01514c6c

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi060 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e216c1ee110397eb48ef026c794021917db737f2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6379624]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:17:43
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379624/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379624/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d438243adcc84ee69e4bef2d99641878

[6379625]  rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:15:15
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379625/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379625/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b79e83ce8404f6b92d78aa0ff6bdb73

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi028 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e216c1ee110397eb48ef026c794021917db737f2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6379626]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:17:16
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379626/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379626/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22e57ffbbafc4c9dacbecef186779c40

[6379627]  rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:15:02
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379627/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379627/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c41eb8f8c09b4feaa8dab331c3caadd5

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi039 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e216c1ee110397eb48ef026c794021917db737f2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6379628]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:17:16
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379628/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379628/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e73654a44287465488d292decc5a302d

[6379629]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:19:08
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379629/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379629/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13e51fa56b43460f8a29fe4ac9fbb826

    reached maximum tries (90) after waiting for 540 seconds

[6379630]  rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379630/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379630/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5fa5f01c15914833850d3b14c1023cd7

    Command failed (workunit test cls/test_cls_rgw.sh) on smithi122 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e216c1ee110397eb48ef026c794021917db737f2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

[6379631]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:20
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379631/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379631/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6bc280640de94d1f92e8496ac60e21dc

[6379632]  rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:20:42
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379632/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379632/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9448be62f5114fa9bc81ef8442efdcbc

    "2021-09-08T04:19:23.194012+0000 mon.a (mon.0) 117 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6379633]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:28
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379633/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379633/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c9cf0a3ef4954012a94d15a456206c65

[6379635]  rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:16:17
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379635/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379635/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0bf0ebb9ac0b43aa98baeafa10a386ce



Pass
=================================================================
[6379621] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:26:27
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379621/

[6379634] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:23:22
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379634/

[6379636] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:12:02
info:   http://pulpito.ceph.com/ivancich-2021-09-08_03:58:43-rgw-master-distro-basic-smithi/6379636/