2021-08-25T04:02:04.425 INFO:root:teuthology version: 1.1.0-d0ccb5e7
2021-08-25T04:02:04.426 INFO:teuthology.results:subset: '1/3'
2021-08-25T04:02:04.426 INFO:teuthology.results:seed: '831'
2021-08-25T04:15:02.644 INFO:root:teuthology version: 1.1.0-d0ccb5e7
2021-08-25T04:15:02.673 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-08-25T05:50:09.852 INFO:teuthology.results:Tests finished! gathering results...
2021-08-25T05:50:10.959 INFO:scrape:Found 16 jobs
2021-08-25T05:50:12.184 INFO:scrape:Found 2 distinct failure reasons
2021-08-25T05:50:12.184 INFO:scrape:Failure: "2021-08-25T04:35:58.256942+0000 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
2021-08-25T05:50:12.185 INFO:scrape:3 jobs: ['6358293', '6358292', '6358297']
2021-08-25T05:50:12.185 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls']
2021-08-25T05:50:12.186 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-25T05:50:12.186 INFO:scrape:
2021-08-25T05:50:12.187 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2021-08-25T05:50:12.187 INFO:scrape:2 jobs: ['6358290', '6358303']
2021-08-25T05:50:12.188 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'sharding$/{default}', 'tasks/{cls']
2021-08-25T05:50:12.188 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2021-08-25T05:50:12.189 INFO:scrape:
2021-08-25T05:50:12.190 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  11


Fail
=================================================================
[6358290]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:18:06
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358290/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358290/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7089644002d947eaa03988fbb00dfa9f

    reached maximum tries (90) after waiting for 540 seconds

[6358292]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:27:55
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358292/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358292/

    "2021-08-25T04:23:53.685803+0000 mon.a (mon.0) 132 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6358293]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:17:51
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358293/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358293/

    "2021-08-25T04:35:58.256942+0000 mon.a (mon.0) 503 : cluster [WRN] Health
    check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in
    cluster log

[6358297]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:19:58
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358297/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358297/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f006652b0a264465a28160c260ee69be

    "2021-08-25T04:30:05.824839+0000 mon.a (mon.0) 134 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6358303]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:19:45
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358303/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358303/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97fe17dd5c914983a220af720207f81b

    reached maximum tries (90) after waiting for 540 seconds



Pass
=================================================================
[6358288] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:25:48
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358288/

[6358289] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:25:46
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358289/

[6358291] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:11:49
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358291/

[6358294] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:27:21
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358294/

[6358295] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:17:08
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358295/

[6358296] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:26:48
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358296/

[6358298] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:27:20
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358298/

[6358299] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:27:30
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358299/

[6358300] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:27:58
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358300/

[6358301] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:26:48
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358301/

[6358302] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:29:04
info:   http://pulpito.ceph.com/ivancich-2021-08-25_03:48:11-rgw-wip-fix-bi-test-error-distro-basic-smithi/6358302/