2023-04-11T07:53:32.740 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-11T07:53:32.741 INFO:teuthology.results:subset: '1/3'
2023-04-11T07:53:32.741 INFO:teuthology.results:seed: '2404'
2023-04-11T07:54:40.913 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-11T07:54:40.917 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-11T19:55:33.745 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-04-11T19:55:37.232 INFO:scrape:Found 48 jobs
2023-04-11T19:55:37.234 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/soumyakoduri-2023-04-11_07:53:20-rgw-wip-skoduri-archive-distro-default-smithi/7238091/teuthology.log
2023-04-11T19:55:38.046 WARNING:scrape:Not-understood crash indication 2023-04-11T10:48:58.924 INFO:tasks.rgw.c1.client.1.smithi023.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.046 WARNING:scrape:Not-understood crash indication 2023-04-11T18:41:15.030 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi023.stderr: INFO: daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.047 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:38.256 WARNING:scrape:Not-understood crash indication 2023-04-11T10:48:58.924 INFO:tasks.rgw.c1.client.1.smithi023.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.257 WARNING:scrape:Not-understood crash indication 2023-04-11T18:41:15.030 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi023.stderr: INFO: daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.257 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:38.460 WARNING:scrape:Not-understood crash indication 2023-04-11T10:48:58.924 INFO:tasks.rgw.c1.client.1.smithi023.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.461 WARNING:scrape:Not-understood crash indication 2023-04-11T18:41:15.030 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi023.stderr: INFO: daemon-helper: command crashed with signal 11
2023-04-11T19:55:38.461 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:39.263 WARNING:scrape:Not-understood crash indication 2023-04-11T08:13:55.357 INFO:tasks.rgw.client.0.smithi033.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:39.264 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:39.337 WARNING:scrape:Not-understood crash indication 2023-04-11T08:13:55.357 INFO:tasks.rgw.client.0.smithi033.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:39.338 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:39.413 WARNING:scrape:Not-understood crash indication 2023-04-11T08:13:55.357 INFO:tasks.rgw.client.0.smithi033.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:39.414 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:39.489 WARNING:scrape:Not-understood crash indication 2023-04-11T08:13:55.357 INFO:tasks.rgw.client.0.smithi033.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:39.490 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:39.562 WARNING:scrape:Not-understood crash indication 2023-04-11T08:13:55.357 INFO:tasks.rgw.client.0.smithi033.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:39.563 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:41.435 WARNING:scrape:Not-understood crash indication 2023-04-11T10:00:27.398 INFO:tasks.rgw.client.0.smithi186.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:41.436 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:41.525 WARNING:scrape:Not-understood crash indication 2023-04-11T10:00:27.398 INFO:tasks.rgw.client.0.smithi186.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:41.526 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:41.613 WARNING:scrape:Not-understood crash indication 2023-04-11T10:00:27.398 INFO:tasks.rgw.client.0.smithi186.stderr:daemon-helper: command crashed with signal 11
2023-04-11T19:55:41.614 WARNING:scrape:Not-understood crash indication 
2023-04-11T19:55:42.446 INFO:scrape:Found 7 distinct failure reasons
2023-04-11T19:55:42.447 INFO:scrape:Dead: None
2023-04-11T19:55:42.448 INFO:scrape:9 jobs: ['7238091', '7238083', '7238026', '7238022', '7238032', '7238067', '7238089', '7238014', '7238100']
2023-04-11T19:55:42.448 INFO:scrape:
2023-04-11T19:55:42.448 INFO:scrape:Failure: rgw multisite test failures
2023-04-11T19:55:42.449 INFO:scrape:2 jobs: ['7238016', '7238069']
2023-04-11T19:55:42.449 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2023-04-11T19:55:42.449 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2023-04-11T19:55:42.450 INFO:scrape:
2023-04-11T19:55:42.450 INFO:scrape:Failure: Command failed (workunit test rgw/test_librgw_file.sh) on smithi073 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=89852674f1460911903180149f15607148fff4fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-04-11T19:55:42.450 INFO:scrape:7 jobs: ['7238075', '7238034', '7238059', '7238040', '7238051', '7238079', '7238098']
2023-04-11T19:55:42.451 INFO:scrape:suites intersection: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'tasks/{cls']
2023-04-11T19:55:42.451 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2023-04-11T19:55:42.451 INFO:scrape:
2023-04-11T19:55:42.451 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi033 with status 1: 'source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m cloud_transition'
2023-04-11T19:55:42.452 INFO:scrape:2 jobs: ['7238004', '7238012']
2023-04-11T19:55:42.452 INFO:scrape:suites intersection: ['overrides', 's3tests-branch']
2023-04-11T19:55:42.452 INFO:scrape:suites union: ['overrides', 'rgw/cloud-transition/{cluster', 'rgw/lifecycle/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_s3tests}']
2023-04-11T19:55:42.453 INFO:scrape:
2023-04-11T19:55:42.453 INFO:scrape:Dead: b"2023-04-11T10:31:15.537 INFO:teuthology.orchestra.run.smithi043.stderr:2023-04-11T10:31:15.537+0000 7f00981b1780 -1 asok(0x55686c366000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.0.150272.asok': (13) Permission denied"
2023-04-11T19:55:42.453 INFO:scrape:2 jobs: ['7238049', '7238073']
2023-04-11T19:55:42.454 INFO:scrape:suites intersection: ['2-setup', 'cluster', 'frontend/beast', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/quincy/{distro$/{centos_latest}']
2023-04-11T19:55:42.454 INFO:scrape:suites union: ['2-setup', '3-upgrade-sequence/osds-then-rgws', '3-upgrade-sequence/rgws-then-osds', 'cluster', 'frontend/beast', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/quincy/{distro$/{centos_latest}']
2023-04-11T19:55:42.454 INFO:scrape:
2023-04-11T19:55:42.455 INFO:scrape:Failure: Command failed on smithi038 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-04-11T19:55:42.455 INFO:scrape:1 jobs: ['7238042']
2023-04-11T19:55:42.455 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{ubuntu_latest}}']
2023-04-11T19:55:42.455 INFO:scrape:
2023-04-11T19:55:42.456 INFO:scrape:Failure: Cannot connect to remote host smithi179
2023-04-11T19:55:42.456 INFO:scrape:1 jobs: ['7238085']
2023-04-11T19:55:42.456 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/replicated', 's3tests-branch', 'tasks/rgw_ragweed', 'ubuntu_latest}']
2023-04-11T19:55:42.456 INFO:scrape: