2023-03-03T18:24:44.976 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-03T18:24:44.977 INFO:teuthology.results:subset: '111/120000'
2023-03-03T18:24:44.977 INFO:teuthology.results:seed: '7622'
2023-03-03T19:20:53.414 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-03T19:20:53.432 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-04T05:11:42.305 INFO:teuthology.results:Tests finished! gathering results...
2023-03-04T05:11:44.980 INFO:scrape:Found 37 jobs
2023-03-04T05:11:48.064 WARNING:scrape:Not-understood crash indication 2023-03-03T20:23:05.685 INFO:tasks.rgw.c1.client.0.smithi035.stderr:daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.065 WARNING:scrape:Not-understood crash indication 2023-03-04T04:09:41.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi035.stderr: INFO: daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.065 WARNING:scrape:Not-understood crash indication 
2023-03-04T05:11:48.249 WARNING:scrape:Not-understood crash indication 2023-03-03T20:23:05.685 INFO:tasks.rgw.c1.client.0.smithi035.stderr:daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.250 WARNING:scrape:Not-understood crash indication 2023-03-04T04:09:41.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi035.stderr: INFO: daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.250 WARNING:scrape:Not-understood crash indication 
2023-03-04T05:11:48.451 WARNING:scrape:Not-understood crash indication 2023-03-03T20:23:05.685 INFO:tasks.rgw.c1.client.0.smithi035.stderr:daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.452 WARNING:scrape:Not-understood crash indication 2023-03-04T04:09:41.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi035.stderr: INFO: daemon-helper: command crashed with signal 6
2023-03-04T05:11:48.452 WARNING:scrape:Not-understood crash indication 
2023-03-04T05:11:48.453 INFO:scrape:Found 3 distinct failure reasons
2023-03-04T05:11:48.453 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi116 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not fails_on_dbstore'"
2023-03-04T05:11:48.453 INFO:scrape:3 jobs: ['7192564', '7192570', '7192561']
2023-03-04T05:11:48.454 INFO:scrape:suites intersection: ['overrides', 's3tests-branch']
2023-03-04T05:11:48.454 INFO:scrape:suites union: ['overrides', 'rgw/cloud-transition/{cluster', 'rgw/dbstore/{cluster', 'rgw/lifecycle/{cluster', 's3tests-branch', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_s3tests}']
2023-03-04T05:11:48.454 INFO:scrape:
2023-03-04T05:11:48.454 INFO:scrape:Failure: Command failed on smithi187 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'
2023-03-04T05:11:48.455 INFO:scrape:1 jobs: ['7192591']
2023-03-04T05:11:48.455 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-03-04T05:11:48.455 INFO:scrape:
2023-03-04T05:11:48.456 INFO:scrape:Failure: rgw multisite test failures
2023-03-04T05:11:48.456 INFO:scrape:2 jobs: ['7192543', '7192576']
2023-03-04T05:11:48.456 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-03-04T05:11:48.456 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-03-04T05:11:48.457 INFO:scrape:
2023-03-04T05:11:48.459 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  31


Fail
=================================================================
[7192543]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   01:54:05
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192543/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192543/
sentry: https://sentry.ceph.com/organizations/ceph/?query=87e904845c10477f89d264cb45c6c7cc

    rgw multisite test failures

[7192561]  rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/cloud_transition_s3tests}
-----------------------------------------------------------------
time:   00:32:23
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192561/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192561/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cef69bf04ef454e996ad05d14f4ecdd

    Command failed (s3 tests against rgw) on smithi019 with status 1: 'source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m
    cloud_transition'

[7192564]  rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
-----------------------------------------------------------------
time:   00:20:05
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192564/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192564/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5d640fdf79242dba7f7383d6b87fbba

    Command failed (s3 tests against rgw) on smithi116 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not fails_on_dbstore'"

[7192570]  rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
-----------------------------------------------------------------
time:   00:24:18
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192570/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192570/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e4d229f9c1c42709d481533043d408b

    Command failed (s3 tests against rgw) on smithi099 with status 1: 'source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m
    lifecycle'

[7192576]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   10:12:15
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192576/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192576/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0980463520354013846be014d0157888

    rgw multisite test failures

[7192591]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192591/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192591/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9f7ce2b025c242d89e3a066fddca03f1

    Command failed on smithi187 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'



Pass
=================================================================
[7192532] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:57:49
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192532/

[7192533] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:17:20
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192533/

[7192534] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:10:57
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192534/

[7192535] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:14:30
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192535/

[7192537] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:10:24
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192537/

[7192539] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:14:04
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192539/

[7192540] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:15:50
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192540/

[7192545] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:03
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192545/

[7192547] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:33:51
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192547/

[7192549] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:10:04
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192549/

[7192551] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:17:20
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192551/

[7192553] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}}
time:   00:11:33
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192553/

[7192555] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:15:25
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192555/

[7192557] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:13:42
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192557/

[7192558] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:49:02
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192558/

[7192559] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:16:35
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192559/

[7192560] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:10:41
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192560/

[7192562] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:22:58
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192562/

[7192567] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192567/

[7192573] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:11:49
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192573/

[7192579] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:52:40
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192579/

[7192582] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:14:46
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192582/

[7192585] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:11:37
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192585/

[7192588] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}}
time:   00:28:14
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192588/

[7192594] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192594/

[7192597] rgw/tools/{centos_latest cluster tasks}
time:   02:48:35
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192597/

[7192599] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:35:24
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192599/

[7192601] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:18:23
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192601/

[7192603] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:10:36
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192603/

[7192605] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:17:01
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192605/

[7192607] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:13:55
info:   http://pulpito.ceph.com/yuriw-2023-03-03_17:39:48-rgw-reef-distro-default-smithi/7192607/