2023-01-13T16:27:45.528 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-13T16:27:45.529 INFO:teuthology.results:subset: '111/120000'
2023-01-13T16:27:45.529 INFO:teuthology.results:seed: '5887'
2023-01-13T17:58:48.284 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-13T17:58:48.323 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-01-13T19:56:57.693 INFO:teuthology.results:Tests finished! gathering results...
2023-01-13T19:57:00.689 INFO:scrape:Found 43 jobs
2023-01-13T19:57:02.444 WARNING:scrape:Not-understood crash indication 2023-01-13T17:30:00.220 INFO:tasks.rgw.c1.client.0.smithi164.stderr:daemon-helper: command crashed with signal 6
2023-01-13T19:57:02.445 WARNING:scrape:Not-understood crash indication 2023-01-13T18:37:09.135 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi164.stderr: INFO: daemon-helper: command crashed with signal 6
2023-01-13T19:57:02.445 WARNING:scrape:Not-understood crash indication 
2023-01-13T19:57:02.594 WARNING:scrape:Not-understood crash indication 2023-01-13T17:30:00.220 INFO:tasks.rgw.c1.client.0.smithi164.stderr:daemon-helper: command crashed with signal 6
2023-01-13T19:57:02.595 WARNING:scrape:Not-understood crash indication 2023-01-13T18:37:09.135 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi164.stderr: INFO: daemon-helper: command crashed with signal 6
2023-01-13T19:57:02.595 WARNING:scrape:Not-understood crash indication 
2023-01-13T19:57:02.597 INFO:scrape:Found 3 distinct failure reasons
2023-01-13T19:57:02.598 INFO:scrape:Failure: Command failed on smithi188 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'
2023-01-13T19:57:02.598 INFO:scrape:1 jobs: ['7138918']
2023-01-13T19:57:02.598 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-01-13T19:57:02.599 INFO:scrape:
2023-01-13T19:57:02.599 INFO:scrape:Failure: rgw multisite test failures
2023-01-13T19:57:02.600 INFO:scrape:2 jobs: ['7138936', '7138913']
2023-01-13T19:57:02.600 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-01-13T19:57:02.601 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-01-13T19:57:02.601 INFO:scrape:
2023-01-13T19:57:02.602 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi182 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test'
2023-01-13T19:57:02.603 INFO:scrape:1 jobs: ['7138914']
2023-01-13T19:57:02.603 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2023-01-13T19:57:02.603 INFO:scrape:
2023-01-13T19:57:02.605 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  39


Fail
=================================================================
[7138913]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:01:16
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138913/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=575319c25aa14b4c92632809e8161b0a

    rgw multisite test failures

[7138914]  rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
-----------------------------------------------------------------
time:   01:42:30
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138914/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2afa8d556239437e8cb288df18f3403e

    Command failed (bucket notification tests against different endpoints) on
    smithi182 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/
    rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s
    rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s
    /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v
    -a kafka_test'

[7138918]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:20:08
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138918/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aa0d9146d1574e75810c4e1674b4b53c

    Command failed on smithi188 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'

[7138936]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:05:02
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138936/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2aefb8fe03c441db8265d2c902dcaff3

    rgw multisite test failures



Pass
=================================================================
[7138907] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests}
time:   00:32:45
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138907/

[7138908] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:21:32
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138908/

[7138909] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:18:20
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138909/

[7138910] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:19:30
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138910/

[7138911] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:28:16
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138911/

[7138912] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:11:07
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138912/

[7138915] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:15:02
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138915/

[7138916] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:14:04
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138916/

[7138917] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}}
time:   00:30:47
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138917/

[7138919] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:14:18
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138919/

[7138920] rgw/tools/{centos_latest cluster tasks}
time:   02:46:42
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138920/

[7138921] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:31:37
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138921/

[7138922] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:18:00
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138922/

[7138923] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:13:07
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138923/

[7138924] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:17:56
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138924/

[7138925] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:10:17
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138925/

[7138926] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:11:29
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138926/

[7138927] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:21:01
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138927/

[7138928] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:00:19
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138928/

[7138929] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:15:37
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138929/

[7138930] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:14:49
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138930/

[7138931] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}}
time:   00:13:57
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138931/

[7138932] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:15:47
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138932/

[7138933] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:15:48
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138933/

[7138934] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:15:36
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138934/

[7138935] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:09:42
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138935/

[7138937] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:11:28
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138937/

[7138938] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:13:56
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138938/

[7138939] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:31:10
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138939/

[7138940] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:14:02
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138940/

[7138941] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:13:17
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138941/

[7138942] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:12:38
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138942/

[7138943] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:15:35
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138943/

[7138944] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:17:39
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138944/

[7138945] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:00:10
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138945/

[7138946] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:16:01
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138946/

[7138947] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:10:05
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138947/

[7138948] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}}
time:   00:11:32
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138948/

[7138949] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:13:54
info:   http://pulpito.ceph.com/yuriw-2023-01-12_20:19:30-rgw-main-distro-default-smithi/7138949/