2024-01-13T01:21:51.963 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-13T01:21:51.963 INFO:teuthology.results:subset: '111/12000'
2024-01-13T01:21:51.963 INFO:teuthology.results:seed: '7529'
2024-01-13T02:10:22.943 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-13T02:10:22.972 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-01-13T06:27:43.721 INFO:teuthology.results:Tests finished! gathering results...
2024-01-13T06:27:48.035 INFO:scrape:Found 43 jobs
2024-01-13T06:27:51.023 INFO:scrape:Found 2 distinct failure reasons
2024-01-13T06:27:51.024 INFO:scrape:Failure: Command failed on smithi124 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.3c deep_scrub'
2024-01-13T06:27:51.024 INFO:scrape:2 jobs: ['7514938', '7514881']
2024-01-13T06:27:51.024 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 's3tests-branch']
2024-01-13T06:27:51.024 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw/thrash/{clusters/fixed-2', 'rgw_pool_type/replicated', 's3tests-branch', 'tasks/rgw_ragweed', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'workload/rgw_user_quota}']
2024-01-13T06:27:51.024 INFO:scrape:
2024-01-13T06:27:51.024 INFO:scrape:Failure: rgw multisite test failures
2024-01-13T06:27:51.024 INFO:scrape:2 jobs: ['7514906', '7514945']
2024-01-13T06:27:51.024 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2024-01-13T06:27:51.025 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2024-01-13T06:27:51.025 INFO:scrape:
2024-01-13T06:27:51.026 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  39


Fail
=================================================================
[7514881]  rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
-----------------------------------------------------------------
time:   00:15:38
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514881/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514881/

    Command failed on smithi008 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 3.14 deep_scrub'

[7514906]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   04:27:50
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514906/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514906/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e390c7215c34afa842c267338a4a7ce

    rgw multisite test failures

[7514938]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
-----------------------------------------------------------------
time:   00:14:21
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514938/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514938/

    Command failed on smithi124 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 3.3c deep_scrub'

[7514945]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   03:08:01
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514945/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=50ac9b1fa16341c3bd8c825b69c32ce3

    rgw multisite test failures



Pass
=================================================================
[7514867] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:20:52
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514867/

[7514869] rgw/upgrade/{1-install/reef/{distro$/{ubuntu_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:15:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514869/

[7514871] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:42:29
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514871/

[7514873] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:20:42
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514873/

[7514875] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:36
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514875/

[7514877] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}}
time:   00:14:06
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514877/

[7514879] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:21:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514879/

[7514882] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:14:57
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514882/

[7514885] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:49:17
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514885/

[7514887] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:21:27
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514887/

[7514889] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:13:05
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514889/

[7514891] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests}
time:   00:27:42
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514891/

[7514893] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:30:33
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514893/

[7514895] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests}
time:   00:09:25
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514895/

[7514897] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:16:37
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514897/

[7514899] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:17:37
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514899/

[7514901] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:36:57
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514901/

[7514903] rgw/lua/{beast bluestore-bitmap fixed-2 overrides supported-distros/{centos_latest} tasks/{0-install test_lua}}
time:   00:11:26
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514903/

[7514904] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:14:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514904/

[7514908] rgw/notifications/{beast bluestore-bitmap centos_latest fixed-2 ignore-pg-availability overrides tasks/{0-install test_amqp test_kafka test_others}}
time:   00:35:34
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514908/

[7514910] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:12:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514910/

[7514912] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_latest}}
time:   00:12:13
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514912/

[7514914] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:33:18
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514914/

[7514916] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3tests ubuntu_latest}
time:   00:31:47
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514916/

[7514918] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:15:22
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514918/

[7514920] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:37:27
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514920/

[7514922] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:14:47
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514922/

[7514924] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:43:10
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514924/

[7514926] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:20:43
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514926/

[7514928] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:05
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514928/

[7514930] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:20:48
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514930/

[7514932] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:14:07
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514932/

[7514934] rgw/upgrade/{1-install/reef/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:13:57
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514934/

[7514936] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:14:55
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514936/

[7514940] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:14:35
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514940/

[7514942] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:21:00
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514942/

[7514943] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:21:51
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514943/

[7514944] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:17:04
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514944/

[7514946] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/tempest ubuntu_latest}
time:   00:22:18
info:   https://pulpito.ceph.com/yuriw-2024-01-12_15:23:49-rgw-main-distro-default-smithi/7514946/