2024-03-04T15:47:03.986 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-04T15:47:03.987 INFO:teuthology.results:seed: '1538'
2024-03-04T16:28:22.615 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-04T16:28:22.635 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-04T19:04:34.311 INFO:teuthology.results:Tests finished! gathering results...
2024-03-04T19:04:38.985 INFO:scrape:Found 73 jobs
2024-03-04T19:04:40.393 WARNING:scrape:Not-understood crash indication 2024-03-04T16:09:02.862 INFO:tasks.rgw.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:40.393 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:40.439 WARNING:scrape:Not-understood crash indication 2024-03-04T16:09:02.862 INFO:tasks.rgw.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:40.440 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:40.487 WARNING:scrape:Not-understood crash indication 2024-03-04T16:09:02.862 INFO:tasks.rgw.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:40.487 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:40.533 WARNING:scrape:Not-understood crash indication 2024-03-04T16:09:02.862 INFO:tasks.rgw.client.0.smithi019.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:40.533 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:41.209 WARNING:scrape:Not-understood crash indication 2024-03-04T16:20:34.479 INFO:tasks.rgw.client.0.smithi144.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:41.209 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:41.309 WARNING:scrape:Not-understood crash indication 2024-03-04T16:20:34.479 INFO:tasks.rgw.client.0.smithi144.stderr:daemon-helper: command crashed with signal 11
2024-03-04T19:04:41.310 WARNING:scrape:Not-understood crash indication 
2024-03-04T19:04:44.902 INFO:scrape:Found 4 distinct failure reasons
2024-03-04T19:04:44.903 INFO:scrape:Failure: Command failed (workunit test rgw/test_librgw_file.sh) on smithi044 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5335f5f0d4a33387826931c0d9be1a7ec43d970 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2024-03-04T19:04:44.903 INFO:scrape:24 jobs: ['7581304', '7581310', '7581314', '7581323', '7581296', '7581283', '7581275', '7581315', '7581325', '7581307', '7581278', '7581319', '7581292', '7581326', '7581280', '7581277', '7581306', '7581288', '7581297', '7581286', '7581328', '7581320', '7581273', '7581293']
2024-03-04T19:04:44.903 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'tasks/{bucket-check', 'versioning}']
2024-03-04T19:04:44.903 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'validater/valgrind}', 'versioning}']
2024-03-04T19:04:44.903 INFO:scrape:
2024-03-04T19:04:44.904 INFO:scrape:Dead: b'2024-03-04T19:04:35.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1'
2024-03-04T19:04:44.904 INFO:scrape:1 jobs: ['7581264']
2024-03-04T19:04:44.904 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}']
2024-03-04T19:04:44.904 INFO:scrape:
2024-03-04T19:04:44.904 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi019 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and cloud_transition'"
2024-03-04T19:04:44.904 INFO:scrape:2 jobs: ['7581256', '7581261']
2024-03-04T19:04:44.904 INFO:scrape:suites intersection: ['ignore-pg-availability', 'overrides', 's3tests-branch']
2024-03-04T19:04:44.904 INFO:scrape:suites union: ['ignore-pg-availability', 'overrides', 'rgw/cloud-transition/{cluster', 'rgw/lifecycle/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_s3tests}']
2024-03-04T19:04:44.904 INFO:scrape:
2024-03-04T19:04:44.904 INFO:scrape:Dead: b'2024-03-04T19:04:40.258 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048476 target=00000000000000000000:00000000000000048336, shard=1 master=00000000000000000000:00000000000000033887 target=00000000000000000000:00000000000000033747, shard=2 master=00000000000000000000:00000000000000055053 target=00000000000000000000:00000000000000054773'
2024-03-04T19:04:44.905 INFO:scrape:1 jobs: ['7581300']
2024-03-04T19:04:44.905 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}']
2024-03-04T19:04:44.905 INFO:scrape: