2026-02-18T12:24:29.798 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-18T12:24:29.798 INFO:teuthology.results:seed: '7375'
2026-02-18T13:02:59.618 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-18T13:02:59.626 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-18T14:25:04.448 INFO:teuthology.results:Tests finished! gathering results...
2026-02-18T14:25:05.507 INFO:scrape:Found 71 jobs
2026-02-18T14:25:05.765 WARNING:scrape:Not-understood crash indication 2026-02-18T12:58:54.853 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:05.765 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:19.718 INFO:tasks.rgw.c1.client.0.trial054.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:05.765 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:26.910 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.017 WARNING:scrape:Not-understood crash indication 2026-02-18T12:58:54.853 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.017 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:19.718 INFO:tasks.rgw.c1.client.0.trial054.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.017 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:26.910 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.270 WARNING:scrape:Not-understood crash indication 2026-02-18T12:58:54.853 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.270 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:19.718 INFO:tasks.rgw.c1.client.0.trial054.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:06.270 WARNING:scrape:Not-understood crash indication 2026-02-18T13:13:26.910 INFO:tasks.rgw.c1.client.1.trial130.stderr:daemon-helper: command crashed with signal 6
2026-02-18T14:25:09.812 WARNING:scrape:Not-understood crash indication 2026-02-18T13:06:26.517 INFO:tasks.rgw.c2.client.0.trial148.stderr:daemon-helper: command crashed with signal 11
2026-02-18T14:25:09.812 WARNING:scrape:Not-understood crash indication 2026-02-18T13:21:06.382 INFO:tasks.rgw.c1.client.0.trial042.stderr:daemon-helper: command crashed with signal 11
2026-02-18T14:25:09.812 WARNING:scrape:Not-understood crash indication 2026-02-18T13:21:19.460 INFO:tasks.rgw.c2.client.0.trial148.stderr:daemon-helper: command crashed with signal 11
2026-02-18T14:25:13.117 INFO:scrape:Found 18 distinct failure reasons
2026-02-18T14:25:13.117 INFO:scrape:Failure: rgw multisite test failures
2026-02-18T14:25:13.117 INFO:scrape:2 jobs: ['56383', '56418']
2026-02-18T14:25:13.117 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2026-02-18T14:25:13.117 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'realms/two-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2026-02-18T14:25:13.117 INFO:scrape:
2026-02-18T14:25:13.117 INFO:scrape:Failure: Command failed (workunit test rgw/s3_user_quota-run.sh) on trial098 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_user_quota-run.sh'
2026-02-18T14:25:13.117 INFO:scrape:3 jobs: ['56429', '56382', '56406']
2026-02-18T14:25:13.117 INFO:scrape:suites intersection: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{0-install', 'rgw_multipart_upload', 'rgw_ragweed', 'rgw_s3tests', 'rgw_user_quota}', 's3tests-branch', 'tasks/{rgw_bucket_quota', 'ubuntu_latest}']
2026-02-18T14:25:13.117 INFO:scrape:suites union: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{0-install', 'rgw_multipart_upload', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'rgw_ragweed', 'rgw_s3tests', 'rgw_user_quota}', 's3tests-branch', 'tasks/{rgw_bucket_quota', 'ubuntu_latest}']
2026-02-18T14:25:13.117 INFO:scrape:
2026-02-18T14:25:13.117 INFO:scrape:Failure: valgrind error: SyscallParam
__libc_sendmsg
sendmsg

2026-02-18T14:25:13.117 INFO:scrape:2 jobs: ['56400', '56426']
2026-02-18T14:25:13.117 INFO:scrape:suites intersection: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-distros/{ubuntu_latest}', 'valgrind}']
2026-02-18T14:25:13.117 INFO:scrape:suites union: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-distros/{ubuntu_latest}', 'tasks/basic/{0-install', 'tasks/kafka/{0-install', 'test_basic}', 'test_kafka}', 'valgrind}']
2026-02-18T14:25:13.117 INFO:scrape:
2026-02-18T14:25:13.117 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f32acf85bf5]
2026-02-18T14:25:13.118 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f32ac43fc30]
 2: /lib64/libc.so.6(+0x8d03c) [0x7f32ac48d03c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f32acf85bf5]
 6: radosgw-admin() [0x832b76]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0xfe) [0xebf5ce]
 8: main()
 9: /lib64/libc.so.6(+0x2a610) [0x7f32ac42a610]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.118 INFO:scrape:1 jobs: ['56428']
2026-02-18T14:25:13.118 INFO:scrape:suites: ['accounts$/{main}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.118 INFO:scrape:
2026-02-18T14:25:13.118 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f3410d85bf5]
2026-02-18T14:25:13.118 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f341003fc30]
 2: /lib64/libc.so.6(+0x8d03c) [0x7f341008d03c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f3410d85bf5]
 6: radosgw-admin() [0x832b76]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0xfe) [0xebf5ce]
 8: main()
 9: /lib64/libc.so.6(+0x2a610) [0x7f341002a610]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.118 INFO:scrape:1 jobs: ['56397']
2026-02-18T14:25:13.118 INFO:scrape:suites: ['accounts$/{tenant}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.118 INFO:scrape:
2026-02-18T14:25:13.118 INFO:scrape:Failure: Command failed on trial070 with status 1: "cd /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate && openstack project create --or-show --os-username admin --os-password ADMIN --os-user-domain-id default --os-project-name admin --os-project-domain-id default --os-identity-api-version 3 --os-auth-url http://trial070.front.sepia.ceph.com:5000/v3 --description 's3tests project' s3tests --debug"
2026-02-18T14:25:13.118 INFO:scrape:4 jobs: ['56388', '56376', '56408', '56431']
2026-02-18T14:25:13.118 INFO:scrape:suites intersection: ['ignore-pg-availability', 'overrides', 's3tests-branch', 'ubuntu_latest}']
2026-02-18T14:25:13.118 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-1', 'frontend/beast', 'ignore-pg-availability', 'overrides', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/tempest/{0-install', 's3tests-branch', 's3tests}', 'tasks/s3/{auth-order/external-local', 'tasks/s3/{auth-order/local-external', 'tasks/tempest', 'ubuntu_latest}']
2026-02-18T14:25:13.118 INFO:scrape:
2026-02-18T14:25:13.118 INFO:scrape:Failure: No module named 'boto.vendored.six.moves'
2026-02-18T14:25:13.118 INFO:scrape:3 jobs: ['56407', '56430', '56386']
2026-02-18T14:25:13.118 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/radosgw-admin']
2026-02-18T14:25:13.118 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}']
2026-02-18T14:25:13.118 INFO:scrape:
2026-02-18T14:25:13.118 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fcde387ff26]
2026-02-18T14:25:13.118 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcde2b98520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fcde387ff26]
 6: radosgw-admin(+0x604bd5) [0x563d29921bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x563d29f995a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcde2b7fd90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.118 INFO:scrape:1 jobs: ['56405']
2026-02-18T14:25:13.118 INFO:scrape:suites: ['accounts$/{main}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.118 INFO:scrape:
2026-02-18T14:25:13.118 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fd8fd1b5f26]
2026-02-18T14:25:13.118 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd8fc4ce520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fd8fd1b5f26]
 6: radosgw-admin(+0x604bd5) [0x56550c0a4bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x56550c71c5a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fd8fc4b5d90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.118 INFO:scrape:1 jobs: ['56433']
2026-02-18T14:25:13.119 INFO:scrape:suites: ['accounts$/{none}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on trial045 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a http_test'
2026-02-18T14:25:13.119 INFO:scrape:3 jobs: ['56413', '56384', '56439']
2026-02-18T14:25:13.119 INFO:scrape:suites intersection: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'valgrind}']
2026-02-18T14:25:13.119 INFO:scrape:suites union: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-distros/{centos_latest}', 'supported-distros/{ubuntu_latest}', 'tasks/amqp/{0-install', 'tasks/http/{0-install', 'tasks/kafka_failover/{0-install', 'test_amqp}', 'test_http}', 'test_kafka}', 'valgrind}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7f122bb27f26]
2026-02-18T14:25:13.119 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f122ae40520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7f122bb27f26]
 6: radosgw-admin(+0x604bd5) [0x5621019a5bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x56210201d5a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f122ae27d90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.119 INFO:scrape:1 jobs: ['56437']
2026-02-18T14:25:13.119 INFO:scrape:suites: ['accounts$/{main-tenant}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f34d9385bf5]
2026-02-18T14:25:13.119 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f34d883fc30]
 2: /lib64/libc.so.6(+0x8d03c) [0x7f34d888d03c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f34d9385bf5]
 6: radosgw-admin() [0x832b76]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0xfe) [0xebf5ce]
 8: main()
 9: /lib64/libc.so.6(+0x2a610) [0x7f34d882a610]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.119 INFO:scrape:1 jobs: ['56401']
2026-02-18T14:25:13.119 INFO:scrape:suites: ['accounts$/{main-tenant}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off-no-gc}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f19fd185bf5]
2026-02-18T14:25:13.119 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f19fc63fc30]
 2: /lib64/libc.so.6(+0x8d03c) [0x7f19fc68d03c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f19fd185bf5]
 6: radosgw-admin() [0x832b76]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0xfe) [0xebf5ce]
 8: main()
 9: /lib64/libc.so.6(+0x2a610) [0x7f19fc62a610]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.119 INFO:scrape:1 jobs: ['56415']
2026-02-18T14:25:13.119 INFO:scrape:suites: ['accounts$/{main}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7f3af4b84f26]
2026-02-18T14:25:13.119 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3af3e9d520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7f3af4b84f26]
 6: radosgw-admin(+0x604bd5) [0x56203b770bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x56203bde85a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3af3e84d90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.119 INFO:scrape:1 jobs: ['56444']
2026-02-18T14:25:13.119 INFO:scrape:suites: ['accounts$/{main}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off-no-gc}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.119 INFO:scrape:
2026-02-18T14:25:13.119 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fcc102c0f26]
2026-02-18T14:25:13.119 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcc0f5d9520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fcc102c0f26]
 6: radosgw-admin(+0x604bd5) [0x55b226339bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x55b2269b15a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcc0f5c0d90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.120 INFO:scrape:1 jobs: ['56394']
2026-02-18T14:25:13.120 INFO:scrape:suites: ['accounts$/{tenant}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.120 INFO:scrape:
2026-02-18T14:25:13.120 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7fb55e785bf5]
2026-02-18T14:25:13.120 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7fb55dc3fc30]
 2: /lib64/libc.so.6(+0x8d03c) [0x7fb55dc8d03c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7fb55e785bf5]
 6: radosgw-admin() [0x832b76]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0xfe) [0xebf5ce]
 8: main()
 9: /lib64/libc.so.6(+0x2a610) [0x7fb55dc2a610]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.120 INFO:scrape:1 jobs: ['56411']
2026-02-18T14:25:13.120 INFO:scrape:suites: ['accounts$/{tenant}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off-no-gc}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.120 INFO:scrape:
2026-02-18T14:25:13.120 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fb9d49d1f26]
2026-02-18T14:25:13.120 INFO:scrape:ceph version 20.3.0-5325-ge9b4a018 (e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0) tentacle (dev - None)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb9d3cea520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7fb9d49d1f26]
 6: radosgw-admin(+0x604bd5) [0x56182a3f5bd5]
 7: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, cls_rgw_reshard_initiator, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x102) [0x56182aa6d5a2]
 8: main()
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb9d3cd1d90]
 10: __libc_start_main()
 11: _start()
2026-02-18T14:25:13.120 INFO:scrape:1 jobs: ['56423']
2026-02-18T14:25:13.120 INFO:scrape:suites: ['accounts$/{none}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.120 INFO:scrape:
2026-02-18T14:25:13.120 INFO:scrape:Failure: Command failed (workunit test rgw/test_awssdkv4_sig.sh) on trial075 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_awssdkv4_sig.sh'
2026-02-18T14:25:13.120 INFO:scrape:1 jobs: ['56442']
2026-02-18T14:25:13.120 INFO:scrape:suites: ['accounts$/{none}', 'bucket-check', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'rados-pool-quota', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{admin-pagination', 'validater/valgrind}', 'versioning', 'zzz-s3tests-java}']
2026-02-18T14:25:13.120 INFO:scrape:
2026-02-18T14:25:13.122 INFO:teuthology.results:Sending results to anrao@redhat.com: Test Run: anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial
=================================================================
info:    https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/
logs:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/
failed:  29
dead:    0
running: 0
waiting: 0
queued:  0
passed:  42


Fail
=================================================================
[56376]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:05:19
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56376/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56376/

    Command failed on trial170 with status 1: "cd
    /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate &&
    openstack project create --or-show --os-username admin --os-password ADMIN
    --os-user-domain-id default --os-project-name admin --os-project-domain-id
    default --os-identity-api-version 3 --os-auth-url
    http://trial170.front.sepia.ceph.com:5000/v3 --description 'Encryption
    Tenant' --domain default rgwcrypt --debug"

[56382]  rgw/multifs/{0-install clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/{rgw_bucket_quota rgw_multipart_upload rgw_ragweed rgw_s3tests rgw_user_quota} ubuntu_latest}
-----------------------------------------------------------------
time:   00:18:43
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56382/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56382/

    Command failed (workunit test rgw/s3_user_quota-run.sh) on trial068 with
    status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_user_quota-run.sh'

[56383]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   00:45:48
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56383/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56383/

    rgw multisite test failures

[56384]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/amqp/{0-install centos_latest test_amqp} valgrind}
-----------------------------------------------------------------
time:   00:06:09
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56384/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56384/

    Command failed (bucket notification tests against different endpoints) on
    trial005 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/r
    gw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/sr
    c/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s
    /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v
    -a amqp_test'

[56386]  rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:05:57
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56386/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56386/

    No module named 'boto.vendored.six.moves'

[56388]  rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3/{auth-order/external-local s3tests} ubuntu_latest}
-----------------------------------------------------------------
time:   00:05:31
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56388/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56388/

    Command failed on trial070 with status 1: "cd
    /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate &&
    openstack project create --or-show --os-username admin --os-password ADMIN
    --os-user-domain-id default --os-project-name admin --os-project-domain-id
    default --os-identity-api-version 3 --os-auth-url
    http://trial070.front.sepia.ceph.com:5000/v3 --description 's3tests
    project' s3tests --debug"

[56394]  rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:29:57
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56394/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56394/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56397]  rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:14:26
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56397/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56397/

    valgrind error: SyscallParam sendmsg  AsyncConnection::_try_send(bool)

[56400]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/basic/{0-install supported-distros/{ubuntu_latest} test_basic} valgrind}
-----------------------------------------------------------------
time:   00:09:42
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56400/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56400/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56401]  rgw/verify/{0-install accounts$/{main-tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:13:01
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56401/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56401/

    valgrind error: SyscallParam sendmsg  AsyncConnection::_try_send(bool)

[56405]  rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:02:42
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56405/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56405/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56406]  rgw/multifs/{0-install clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/{rgw_bucket_quota rgw_multipart_upload rgw_ragweed rgw_s3tests rgw_user_quota} ubuntu_latest}
-----------------------------------------------------------------
time:   00:18:24
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56406/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56406/

    Command failed (workunit test rgw/s3_user_quota-run.sh) on trial117 with
    status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_user_quota-run.sh'

[56407]  rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:05:04
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56407/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56407/

    No module named 'boto.vendored.six.moves'

[56408]  rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:04:51
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56408/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56408/

    Command failed on trial101 with status 1: "cd
    /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate &&
    openstack service create --os-username admin --os-password ADMIN --os-user-
    domain-id default --os-project-name admin --os-project-domain-id default
    --os-identity-api-version 3 --os-auth-url
    http://trial101.front.sepia.ceph.com:5000/v3 --description 'Swift Service'
    --name swift object-store --debug"

[56411]  rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   00:53:58
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56411/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56411/

    valgrind error: SyscallParam sendmsg  AsyncConnection::_try_send(bool)

[56413]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/http/{0-install supported-distros/{ubuntu_latest} test_http} valgrind}
-----------------------------------------------------------------
time:   00:09:49
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56413/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56413/

    Command failed (bucket notification tests against different endpoints) on
    trial045 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/r
    gw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/sr
    c/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s
    /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v
    -a http_test'

[56415]  rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   00:53:34
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56415/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56415/

    valgrind error: SyscallParam sendmsg  AsyncConnection::_try_send(bool)

[56418]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   00:37:49
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56418/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56418/

    rgw multisite test failures

[56423]  rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:28:35
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56423/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56423/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56426]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/kafka/{0-install supported-distros/{ubuntu_latest} test_kafka} valgrind}
-----------------------------------------------------------------
time:   00:19:31
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56426/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56426/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56428]  rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:14:32
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56428/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56428/

    valgrind error: SyscallParam sendmsg  AsyncConnection::_try_send(bool)

[56429]  rgw/multifs/{0-install clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/{rgw_bucket_quota rgw_multipart_upload rgw_ragweed rgw_s3tests rgw_user_quota} ubuntu_latest}
-----------------------------------------------------------------
time:   00:18:25
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56429/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56429/

    Command failed (workunit test rgw/s3_user_quota-run.sh) on trial098 with
    status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_user_quota-run.sh'

[56430]  rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:05:18
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56430/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56430/

    No module named 'boto.vendored.six.moves'

[56431]  rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3/{auth-order/local-external s3tests} ubuntu_latest}
-----------------------------------------------------------------
time:   00:04:59
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56431/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56431/

    Command failed on trial101 with status 1: "cd
    /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate &&
    openstack project create --or-show --os-username admin --os-password ADMIN
    --os-user-domain-id default --os-project-name admin --os-project-domain-id
    default --os-identity-api-version 3 --os-auth-url
    http://trial101.front.sepia.ceph.com:5000/v3 --description 's3tests
    project' s3tests --debug"

[56433]  rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:25:10
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56433/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56433/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56437]  rgw/verify/{0-install accounts$/{main-tenant} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   01:01:50
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56437/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56437/

    valgrind error: SyscallParam __libc_sendmsg sendmsg

[56439]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/kafka_failover/{0-install supported-distros/{centos_latest} test_kafka} valgrind}
-----------------------------------------------------------------
time:   00:06:20
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56439/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56439/

    Command failed (bucket notification tests against different endpoints) on
    trial117 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/r
    gw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/sr
    c/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s
    /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v
    -a kafka_failover'

[56442]  rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   00:13:20
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56442/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56442/

    Command failed (workunit test rgw/test_awssdkv4_sig.sh) on trial075 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e9b4a018b34a7b395cbc09ccca25b14ff1f0dfd0
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_awssdkv4_sig.sh'

[56444]  rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/valgrind}
-----------------------------------------------------------------
time:   00:59:47
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56444/
log:    https://qa-proxy.ceph.com/teuthology/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56444/

    valgrind error: SyscallParam __libc_sendmsg sendmsg



Pass
=================================================================
[56374] rgw/bucket-logging/{0-install beast fixed-1 ignore-pg-availability overrides pools/erasure s3tests-branch supported-distros/{ubuntu_latest} tasks/{s3tests}}
time:   00:09:51
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56374/

[56375] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/restore/cloud_restore_s3tests}
time:   00:23:02
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56375/

[56377] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests}
time:   00:05:22
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56377/

[56378] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:06:54
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56378/

[56379] rgw/dedup/{beast bluestore-bitmap fixed-3-rgw ignore-pg-availability overrides supported-distros/{ubuntu_latest} tasks/{0-install test_dedup}}
time:   00:45:35
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56379/

[56380] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:48:04
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56380/

[56381] rgw/lua/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-distros/{ubuntu_latest} tasks/{0-install test_lua}}
time:   00:05:50
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56381/

[56385] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:06:26
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56385/

[56387] rgw/sts/{auth-order/local-sts cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:30:25
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56387/

[56389] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:06:10
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56389/

[56390] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   00:09:10
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56390/

[56391] rgw/upgrade/{1-install/squid/{distro/centos_latest install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:08:15
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56391/

[56392] rgw/verify/{0-install accounts$/{main-tenant} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:34:38
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56392/

[56393] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/s3tests-website}
time:   00:13:09
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56393/

[56395] rgw/verify/{0-install accounts$/{main-tenant} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:32:21
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56395/

[56396] rgw/upgrade/{1-install/tentacle/{distro/centos_latest install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:08:11
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56396/

[56398] rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:31:54
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56398/

[56399] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides s3tests-branch ubuntu_latest}
time:   00:16:21
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56399/

[56402] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:06:38
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56402/

[56403] rgw/upgrade/{1-install/squid/{distro/ubuntu_latest install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:07:53
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56403/

[56404] rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:42:28
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56404/

[56409] rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:43:23
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56409/

[56410] rgw/upgrade/{1-install/tentacle/{distro/ubuntu_latest install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:07:19
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56410/

[56412] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides s3tests-branch ubuntu_latest}
time:   00:11:47
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56412/

[56414] rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:42:36
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56414/

[56416] rgw/bucket-logging/{0-install beast fixed-1 ignore-pg-availability overrides pools/replicated s3tests-branch supported-distros/{centos_latest} tasks/{s3tests}}
time:   00:09:21
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56416/

[56417] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/transition/cloud_transition_s3tests}
time:   00:21:04
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56417/

[56419] rgw/sts/{auth-order/sts-local cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:31:10
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56419/

[56420] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:12:36
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56420/

[56421] rgw/upgrade/{1-install/squid/{distro/centos_latest install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:09:18
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56421/

[56422] rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:35:30
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56422/

[56424] rgw/verify/{0-install accounts$/{main} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:32:54
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56424/

[56425] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides s3tests-branch ubuntu_latest}
time:   00:12:00
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56425/

[56427] rgw/upgrade/{1-install/tentacle/{distro/centos_latest install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:08:07
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56427/

[56432] rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:33:23
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56432/

[56434] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:05:49
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56434/

[56435] rgw/upgrade/{1-install/squid/{distro/ubuntu_latest install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:07:58
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56435/

[56436] rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:43:33
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56436/

[56438] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability overrides s3tests-branch ubuntu_latest}
time:   00:12:25
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56438/

[56440] rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off-no-gc} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:42:31
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56440/

[56441] rgw/upgrade/{1-install/tentacle/{distro/ubuntu_latest install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:06:57
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56441/

[56443] rgw/verify/{0-install accounts$/{main-tenant} clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{admin-pagination bucket-check cls mp_reupload rados-pool-quota ragweed reshard s3tests versioning zzz-s3tests-java} validater/lockdep}
time:   00:43:09
info:   https://pulpito.ceph.com/anuchaithra-2026-02-18_10:16:46-rgw-wip-anrao3-testing-2026-02-18-1215-distro-default-trial/56443/
2026-02-18T14:25:13.131 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 33, in main
    results(args['--archive-dir'], args['--name'], args['--email'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 78, in results
    email_results(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 95, in email_results
    smtp = smtplib.SMTP('localhost')
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection
    sock.connect(sa)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect
    self._internal_connect(address)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused