2023-01-21T12:27:07.233 INFO:root:teuthology version: 0.0.1.dev20+g5abecab
2023-01-21T12:27:07.234 INFO:teuthology.results:subset: '111/120000'
2023-01-21T12:27:07.235 INFO:teuthology.results:seed: '6303'
2023-01-21T13:14:45.722 INFO:root:teuthology version: 0.0.1.dev20+g5abecab
2023-01-21T13:14:45.725 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-01-21T20:05:15.895 INFO:teuthology.results:Tests finished! gathering results...
2023-01-21T20:05:19.404 INFO:scrape:Found 43 jobs
2023-01-21T20:05:19.799 WARNING:scrape:Not-understood crash indication 2023-01-21T13:48:00.617 INFO:tasks.rgw.c1.client.0.smithi040.stderr:daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.800 WARNING:scrape:Not-understood crash indication 2023-01-21T19:57:13.576 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi040.stderr: INFO: daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.800 WARNING:scrape:Not-understood crash indication 
2023-01-21T20:05:19.856 WARNING:scrape:Not-understood crash indication 2023-01-21T13:48:00.617 INFO:tasks.rgw.c1.client.0.smithi040.stderr:daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.858 WARNING:scrape:Not-understood crash indication 2023-01-21T19:57:13.576 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi040.stderr: INFO: daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.858 WARNING:scrape:Not-understood crash indication 
2023-01-21T20:05:19.916 WARNING:scrape:Not-understood crash indication 2023-01-21T13:48:00.617 INFO:tasks.rgw.c1.client.0.smithi040.stderr:daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.917 WARNING:scrape:Not-understood crash indication 2023-01-21T19:57:13.576 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi040.stderr: INFO: daemon-helper: command crashed with signal 6
2023-01-21T20:05:19.918 WARNING:scrape:Not-understood crash indication 
2023-01-21T20:05:21.263 INFO:scrape:Found 2 distinct failure reasons
2023-01-21T20:05:21.264 INFO:scrape:Failure: rgw multisite test failures
2023-01-21T20:05:21.264 INFO:scrape:2 jobs: ['7131676', '7131653']
2023-01-21T20:05:21.265 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-01-21T20:05:21.266 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-01-21T20:05:21.266 INFO:scrape:
2023-01-21T20:05:21.266 INFO:scrape:Failure: Command failed on smithi085 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap'
2023-01-21T20:05:21.267 INFO:scrape:2 jobs: ['7131665', '7131658']
2023-01-21T20:05:21.267 INFO:scrape:suites intersection: ['frontend/beast', 'overrides', 'ubuntu_latest}']
2023-01-21T20:05:21.268 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'objectstore/filestore-xfs', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw/tempest/{clusters/fixed-1', 'rgw_pool_type/ec', 's3tests-branch', 'tasks/rgw_ragweed', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-01-21T20:05:21.268 INFO:scrape:
2023-01-21T20:05:21.270 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  39


Fail
=================================================================
[7131653]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:08:42
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131653/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131653/
sentry: https://sentry.ceph.com/organizations/ceph/?query=49098df0ea164280b4bb67f8ebbb2686

    rgw multisite test failures

[7131658]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:21:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131658/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131658/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a41de516815a4ce0804f0d1e2e701aeb

    Command failed on smithi154 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'

[7131665]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131665/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131665/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f878f77537e746c981cf02093c0af9d5

    Command failed on smithi085 with status 1: 'cd
    /home/ubuntu/cephtest/ragweed && ./bootstrap'

[7131676]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   07:01:34
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131676/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131676/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54a6c6192cbc4b309efc292b2eb72c59

    rgw multisite test failures



Pass
=================================================================
[7131647] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/cloud_transition_s3tests}
time:   00:33:03
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131647/

[7131648] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:24:08
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131648/

[7131649] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:18:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131649/

[7131650] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:18:09
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131650/

[7131651] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:29:25
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131651/

[7131652] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:11:28
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131652/

[7131654] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:48:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131654/

[7131655] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:14:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131655/

[7131656] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131656/

[7131657] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:31:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131657/

[7131659] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131659/

[7131660] rgw/tools/{centos_latest cluster tasks}
time:   02:47:02
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131660/

[7131661] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:30:20
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131661/

[7131662] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:17:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131662/

[7131663] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:11:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131663/

[7131664] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:17:35
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131664/

[7131666] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:13:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131666/

[7131667] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:18:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131667/

[7131668] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:56:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131668/

[7131669] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:12:37
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131669/

[7131670] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:13:43
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131670/

[7131671] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}}
time:   00:10:49
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131671/

[7131672] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:12:16
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131672/

[7131673] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:17:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131673/

[7131674] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:14:51
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131674/

[7131675] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:08:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131675/

[7131677] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:10:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131677/

[7131678] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:13:14
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131678/

[7131679] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:30:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131679/

[7131680] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:12:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131680/

[7131681] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:16:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131681/

[7131682] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:10:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131682/

[7131683] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}}
time:   00:15:09
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131683/

[7131684] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:15:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131684/

[7131685] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:55:49
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131685/

[7131686] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:16:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131686/

[7131687] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:09:00
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131687/

[7131688] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:14:33
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131688/

[7131689] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:13:03
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:32:53-rgw-main-distro-default-smithi/7131689/