2023-12-16T15:37:53.209 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-16T15:37:53.210 INFO:teuthology.results:subset: '111/12000'
2023-12-16T15:37:53.210 INFO:teuthology.results:seed: '4683'
2023-12-16T16:20:15.900 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-16T16:20:15.903 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-16T18:57:28.660 INFO:teuthology.results:Tests finished! gathering results...
2023-12-16T18:57:31.914 INFO:scrape:Found 43 jobs
2023-12-16T18:57:32.115 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.854 INFO:tasks.rgw.c1.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.117 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.945 INFO:tasks.rgw.c2.client.0.smithi146.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.117 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.275 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi019.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.117 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.276 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi146.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.117 WARNING:scrape:Not-understood crash indication 
2023-12-16T18:57:32.143 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.854 INFO:tasks.rgw.c1.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.143 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.945 INFO:tasks.rgw.c2.client.0.smithi146.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.143 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.275 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi019.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.143 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.276 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi146.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.143 WARNING:scrape:Not-understood crash indication 
2023-12-16T18:57:32.169 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.854 INFO:tasks.rgw.c1.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.169 WARNING:scrape:Not-understood crash indication 2023-12-16T16:24:40.945 INFO:tasks.rgw.c2.client.0.smithi146.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.169 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.275 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi019.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.169 WARNING:scrape:Not-understood crash indication 2023-12-16T18:02:03.276 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi146.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.169 WARNING:scrape:Not-understood crash indication 
2023-12-16T18:57:32.795 WARNING:scrape:Not-understood crash indication 2023-12-16T16:41:46.733 INFO:tasks.rgw.c1.client.0.smithi043.stderr:daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.796 WARNING:scrape:Not-understood crash indication 2023-12-16T18:04:26.381 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stderr: INFO: daemon-helper: command crashed with signal 11
2023-12-16T18:57:32.796 WARNING:scrape:Not-understood crash indication 
2023-12-16T18:57:32.925 INFO:scrape:Found 2 distinct failure reasons
2023-12-16T18:57:32.926 INFO:scrape:Failure: rgw multisite test failures
2023-12-16T18:57:32.926 INFO:scrape:2 jobs: ['7494316', '7494337']
2023-12-16T18:57:32.926 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-12-16T18:57:32.926 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-12-16T18:57:32.926 INFO:scrape:
2023-12-16T18:57:32.926 INFO:scrape:Failure: Command failed on smithi046 with status 2: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'
2023-12-16T18:57:32.926 INFO:scrape:1 jobs: ['7494309']
2023-12-16T18:57:32.927 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-12-16T18:57:32.927 INFO:scrape:
2023-12-16T18:57:32.929 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  40


Fail
=================================================================
[7494309]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:14:56
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494309/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494309/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9b70f187d6743b8aad597aa8d50390b

    Command failed on smithi046 with status 2: 'cd
    /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-
    venv/bin/activate && tox -e venv --notest'

[7494316]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:01:14
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494316/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494316/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c30c608bdfe446dba417fb4c8c269e8

    rgw multisite test failures

[7494337]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   01:55:25
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494337/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494337/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0852ee1245e345ada1a333cd8954a83b

    rgw multisite test failures



Pass
=================================================================
[7494296] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:28
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494296/

[7494297] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:14
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494297/

[7494298] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:36:56
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494298/

[7494299] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:48
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494299/

[7494300] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:17:49
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494300/

[7494301] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}}
time:   00:15:57
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494301/

[7494302] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:24:45
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494302/

[7494303] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:15:51
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494303/

[7494304] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:21:48
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494304/

[7494305] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:18:28
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494305/

[7494306] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:26:28
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494306/

[7494307] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:15:12
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494307/

[7494308] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests}
time:   00:30:39
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494308/

[7494310] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests}
time:   00:13:44
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494310/

[7494311] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:16:11
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494311/

[7494312] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:19:23
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494312/

[7494313] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:37:45
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494313/

[7494314] rgw/lua/{beast bluestore-bitmap fixed-2 overrides supported-distros/{ubuntu_latest} tasks/{0-install test_lua}}
time:   00:15:44
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494314/

[7494315] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:16:13
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494315/

[7494317] rgw/notifications/{beast bluestore-bitmap centos_latest fixed-2 ignore-pg-availability overrides tasks/{0-install test_amqp test_kafka test_others}}
time:   00:38:42
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494317/

[7494318] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:17:07
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494318/

[7494319] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_latest}}
time:   00:16:09
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494319/

[7494320] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:36:10
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494320/

[7494321] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3tests ubuntu_latest}
time:   00:36:57
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494321/

[7494322] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:35
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494322/

[7494323] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:41:14
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494323/

[7494324] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:21:01
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494324/

[7494325] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:37:29
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494325/

[7494326] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:24:12
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494326/

[7494327] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:15:29
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494327/

[7494328] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:54
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494328/

[7494329] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:16:32
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494329/

[7494330] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:19:46
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494330/

[7494331] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:18:52
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494331/

[7494332] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:18:05
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494332/

[7494333] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:18:23
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494333/

[7494334] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:26:39
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494334/

[7494335] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:24:57
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494335/

[7494336] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:17:48
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494336/

[7494338] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/tempest ubuntu_latest}
time:   00:25:36
info:   https://pulpito.ceph.com/yuriw-2023-12-16_00:16:31-rgw-main-distro-default-smithi/7494338/