2023-05-20T23:25:41.724 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-20T23:25:41.725 INFO:teuthology.results:subset: '111/12000'
2023-05-20T23:25:41.725 INFO:teuthology.results:seed: '7825'
2023-05-20T23:59:56.716 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-20T23:59:56.740 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-21T09:34:44.231 INFO:teuthology.results:Tests finished! gathering results...
2023-05-21T09:34:47.185 INFO:scrape:Found 41 jobs
2023-05-21T09:34:48.655 WARNING:scrape:Not-understood crash indication 2023-05-21T00:44:01.118 INFO:tasks.rgw.c1.client.0.smithi037.stderr:daemon-helper: command crashed with signal 6
2023-05-21T09:34:48.656 WARNING:scrape:Not-understood crash indication 2023-05-21T08:32:11.968 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi037.stderr: INFO: daemon-helper: command crashed with signal 6
2023-05-21T09:34:48.657 WARNING:scrape:Not-understood crash indication 
2023-05-21T09:34:48.785 INFO:scrape:Found 2 distinct failure reasons
2023-05-21T09:34:48.786 INFO:scrape:Failure: rgw multisite test failures
2023-05-21T09:34:48.787 INFO:scrape:2 jobs: ['7280959', '7280978']
2023-05-21T09:34:48.788 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-05-21T09:34:48.788 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-05-21T09:34:48.788 INFO:scrape:
2023-05-21T09:34:48.789 INFO:scrape:Failure: Command failed on smithi096 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'
2023-05-21T09:34:48.790 INFO:scrape:1 jobs: ['7280983']
2023-05-21T09:34:48.790 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-05-21T09:34:48.790 INFO:scrape:
2023-05-21T09:34:48.792 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7280959]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:05:28
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280959/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280959/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70430d628b4e4a50b852e93f7db4f43b

    rgw multisite test failures

[7280978]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   09:38:51
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280978/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280978/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ff514ea312e49ef9889eacf77ff3e3a

    rgw multisite test failures

[7280983]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:25
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280983/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280983/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4698ee671c34485cb3d322a6b5c863d2

    Command failed on smithi096 with status 1: 'cd
    /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-
    venv/bin/activate && tox -e venv --notest'



Pass
=================================================================
[7280958] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:16:16
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280958/

[7280960] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:37
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280960/

[7280961] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:22:19
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280961/

[7280962] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:34:36
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280962/

[7280963] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:48
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280963/

[7280964] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:14:56
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280964/

[7280965] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:23:11
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280965/

[7280966] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:48
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280966/

[7280967] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:18:28
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280967/

[7280968] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280968/

[7280969] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:49:07
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280969/

[7280970] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:18:27
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280970/

[7280971] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:12:18
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280971/

[7280972] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/cloud_transition_s3tests}
time:   00:33:55
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280972/

[7280973] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:25:11
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280973/

[7280974] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:16:25
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280974/

[7280975] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:17:38
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280975/

[7280976] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:30:37
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280976/

[7280977] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280977/

[7280979] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:53:12
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280979/

[7280980] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:16:44
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280980/

[7280981] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_20.04}}
time:   00:11:54
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280981/

[7280982] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}}
time:   00:29:01
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280982/

[7280984] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:03
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280984/

[7280985] rgw/tools/{centos_latest cluster tasks}
time:   02:51:18
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280985/

[7280986] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:54
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280986/

[7280987] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:32:26
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280987/

[7280988] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:42
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280988/

[7280989] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:02
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280989/

[7280990] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:20:02
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280990/

[7280991] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:15:33
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280991/

[7280992] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:34
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280992/

[7280993] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:59:39
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280993/

[7280994] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:00
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280994/

[7280995] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:15:05
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280995/

[7280996] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}}
time:   00:16:24
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280996/

[7280997] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:24
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280997/

[7280998] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:18:20
info:   http://pulpito.ceph.com/yuriw-2023-05-20_14:41:38-rgw-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7280998/