2023-10-12T00:01:39.481 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07
2023-10-12T00:01:39.483 INFO:teuthology.results:seed: '7936'
2023-10-12T01:01:49.270 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07
2023-10-12T01:01:49.297 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-10-12T03:27:01.258 INFO:teuthology.results:Tests finished! gathering results...
2023-10-12T03:27:07.005 INFO:scrape:Found 76 jobs
2023-10-12T03:27:09.310 INFO:scrape:Found 5 distinct failure reasons
2023-10-12T03:27:09.311 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on smithi071 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/ragweed.client.0 && RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox --sitepackages -- -v -m 'not fails_on_rgw'"
2023-10-12T03:27:09.312 INFO:scrape:1 jobs: ['7422094']
2023-10-12T03:27:09.312 INFO:scrape:suites: ['2-setup', '3-upgrade-sequence/osds-then-rgws', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream}']
2023-10-12T03:27:09.312 INFO:scrape:
2023-10-12T03:27:09.313 INFO:scrape:Failure: Command failed on smithi029 with status 1: 'sudo chmod 777 /var/log/ceph'
2023-10-12T03:27:09.313 INFO:scrape:6 jobs: ['7422119', '7422151', '7422087', '7422106', '7422103', '7422139']
2023-10-12T03:27:09.314 INFO:scrape:suites intersection: ['ignore-pg-availability']
2023-10-12T03:27:09.314 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/kmip', '2-setup', '3-rgw/rgw', '3-upgrade-sequence/osds-then-rgws', '3-upgrade-sequence/rgws-then-osds', '4-tests/{s3tests}', 'bluestore-bitmap', 'centos_latest', 'cluster', 'fixed-2', 'frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'overrides}', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/notifications/{beast', 'rgw/upgrade/{1-install/reef/distro$/{centos_latest}', 'rgw/upgrade/{1-install/reef/distro$/{ubuntu_latest}', 'rgw/upgrade/{1-install/reef/overrides', 's3tests-branch', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}', 'ubuntu_latest}']
2023-10-12T03:27:09.314 INFO:scrape:
2023-10-12T03:27:09.315 INFO:scrape:Failure: An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch.
2023-10-12T03:27:09.315 INFO:scrape:2 jobs: ['7422146', '7422112']
2023-10-12T03:27:09.316 INFO:scrape:suites intersection: ['2-setup', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/reef/install']
2023-10-12T03:27:09.316 INFO:scrape:suites union: ['2-setup', '3-upgrade-sequence/osds-then-rgws', '3-upgrade-sequence/rgws-then-osds', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/reef/install']
2023-10-12T03:27:09.316 INFO:scrape:
2023-10-12T03:27:09.317 INFO:scrape:Failure: rgw multisite test failures
2023-10-12T03:27:09.317 INFO:scrape:2 jobs: ['7422125', '7422086']
2023-10-12T03:27:09.318 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-10-12T03:27:09.318 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-10-12T03:27:09.319 INFO:scrape:
2023-10-12T03:27:09.319 INFO:scrape:Failure: "2023-10-12T01:10:00.000167+0000 mon.a (mon.0) 199 : cluster [WRN] overall HEALTH_WARN Degraded data redundancy: 4/1227 objects degraded (0.326%), 1 pg degraded" in cluster log
2023-10-12T03:27:09.320 INFO:scrape:1 jobs: ['7422136']
2023-10-12T03:27:09.320 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 's3tests-branch', 'tasks/rgw_bucket_quota', 'ubuntu_latest}']
2023-10-12T03:27:09.321 INFO:scrape: