2023-03-06T21:43:33.977 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T21:43:33.978 INFO:teuthology.results:subset: '0/30000' 2023-03-06T21:43:33.978 INFO:teuthology.results:seed: '5519' 2023-03-06T22:22:11.385 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T22:22:11.388 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-07T10:23:05.502 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-07T10:23:08.154 INFO:scrape:Found 37 jobs 2023-03-07T10:23:13.889 INFO:scrape:Found 9 distinct failure reasons 2023-03-07T10:23:13.890 INFO:scrape:Crash: Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' 2023-03-07T10:23:13.890 INFO:scrape:ceph version 18.0.0-2778-g4d310ef8 (4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fb5b58f8cf0] 2: /lib64/librados.so.2(+0xc4734) [0x7fb5b8d53734] 3: (RGWSI_ZoneUtils::init_unique_trans_id_deps()+0x4e) [0x5593f1ad851e] 4: (RGWSI_ZoneUtils::do_start(optional_yield, DoutPrefixProvider const*)+0xd) [0x5593f1ad863d] 5: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x5593f194eab0] 6: (RGWServices_Def::init(ceph::common::CephContext*, bool, bool, bool, librados::v14_2_0::Rados*, optional_yield, DoutPrefixProvider const*)+0x175a) [0x5593f195137a] 7: (RGWServices::do_init(ceph::common::CephContext*, bool, bool, bool, librados::v14_2_0::Rados*, optional_yield, DoutPrefixProvider const*)+0x34) [0x5593f19527f4] 8: (RGWRados::init_svc(bool, DoutPrefixProvider const*)+0x49) [0x5593f18863d9] 9: (DriverManager::init_raw_storage_provider(DoutPrefixProvider const*, ceph::common::CephContext*, DriverManager::Config const&)+0x103) [0x5593f168fdf3] 10: main() 11: __libc_start_main() 12: _start() 2023-03-07T10:23:13.890 INFO:scrape:24 jobs: ['7195470', '7195459', '7195473', '7195477', '7195479', '7195457', '7195460', '7195481', '7195484', '7195453', '7195458', '7195463', '7195485', '7195483', '7195472', '7195489', '7195474', '7195482', '7195467', '7195454', '7195476', '7195475', '7195469', '7195462'] 2023-03-07T10:23:13.891 INFO:scrape:suites intersection: [] 2023-03-07T10:23:13.891 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '2-kms/kmip', '2-kms/testing', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'bluestore-bitmap', 'centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'first', 'fixed-2', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'notify', 'objectstore', 'objectstore/bluestore-bitmap', 'omap_limits', 'overrides', 'pool-type', 'proto/http', 'proto/https', 'ragweed', 'realms/three-zones', 'realms/two-zonegroup', 'reshard', 'rgw/cloud-transition/{cluster', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/lifecycle/{cluster', 'rgw/multifs/{clusters/fixed-2', 'rgw/multisite/{clusters', 'rgw/notifications/{beast', 'rgw/singleton/{all/radosgw-admin', 'rgw/sts/{cluster', 'rgw/verify/{0-install', 'rgw_frontend/beast', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'ststests}}', 'supported-all-distro$/{centos_8}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_bucket_quota', 'tasks/rgw_multipart_upload', 'tasks/rgw_ragweed', 'tasks/rgw_s3tests', 'tasks/rgw_s3tests}', 'tasks/rgw_user_quota', 'tasks/test_multi}', 'tasks/{0-install', 'tasks/{cls', 'test_amqp', 'test_kafka', 'test_others}}', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}'] 2023-03-07T10:23:13.891 INFO:scrape: 2023-03-07T10:23:13.892 INFO:scrape:Dead: b'2023-03-07T10:04:39.383 DEBUG:teuthology.exit:Finished running handlers' 2023-03-07T10:23:13.892 INFO:scrape:1 jobs: ['7195468'] 2023-03-07T10:23:13.892 INFO:scrape:suites: ['frontend/beast', 'http', 'overrides', 'rgw/website/{clusters/fixed-2', 's3tests-branch', 'tasks/s3tests-website', 'ubuntu_latest}'] 2023-03-07T10:23:13.893 INFO:scrape: 2023-03-07T10:23:13.893 INFO:scrape:Failure: Command failed (workunit test rgw/s3_bucket_quota.pl) on smithi017 with status 17: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_bucket_quota.pl' 2023-03-07T10:23:13.893 INFO:scrape:1 jobs: ['7195465'] 2023-03-07T10:23:13.893 INFO:scrape:suites: ['frontend/beast', 'install', 'objectstore/bluestore-bitmap', 'rgw/thrash/{clusters/fixed-2', 's3tests-branch', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'workload/rgw_bucket_quota}'] 2023-03-07T10:23:13.894 INFO:scrape: 2023-03-07T10:23:13.894 INFO:scrape:Failure: Command failed (workunit test rgw/s3_multipart_upload.pl) on smithi120 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_multipart_upload.pl' 2023-03-07T10:23:13.894 INFO:scrape:1 jobs: ['7195471'] 2023-03-07T10:23:13.895 INFO:scrape:suites: ['frontend/beast', 'install', 'objectstore/bluestore-bitmap', 'rgw/thrash/{clusters/fixed-2', 's3tests-branch', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'workload/rgw_multipart_upload}'] 2023-03-07T10:23:13.895 INFO:scrape: 2023-03-07T10:23:13.895 INFO:scrape:Failure: Command failed on smithi042 with status 100: 'sudo apt-get clean' 2023-03-07T10:23:13.895 INFO:scrape:1 jobs: ['7195488'] 2023-03-07T10:23:13.896 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec-profile', 's3tests-branch', 'tasks/rgw_multipart_upload', 'ubuntu_latest}'] 2023-03-07T10:23:13.896 INFO:scrape: 2023-03-07T10:23:13.896 INFO:scrape:Failure: Command failed on smithi053 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' 2023-03-07T10:23:13.897 INFO:scrape:1 jobs: ['7195480'] 2023-03-07T10:23:13.897 INFO:scrape:suites: ['frontend/beast', 'install', 'objectstore/bluestore-bitmap', 'rgw/thrash/{clusters/fixed-2', 's3tests-branch', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'workload/rgw_s3tests}'] 2023-03-07T10:23:13.897 INFO:scrape: 2023-03-07T10:23:13.897 INFO:scrape:Failure: Command failed (workunit test rgw/test_rgw_orphan_list.sh) on smithi196 with status 12: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_orphan_list.sh' 2023-03-07T10:23:13.898 INFO:scrape:1 jobs: ['7195466'] 2023-03-07T10:23:13.898 INFO:scrape:suites: ['cluster', 'rgw/tools/{centos_latest', 'tasks}'] 2023-03-07T10:23:13.898 INFO:scrape: 2023-03-07T10:23:13.899 INFO:scrape:Dead: b"TypeError: argument of type 'NoneType' is not iterable" 2023-03-07T10:23:13.899 INFO:scrape:1 jobs: ['7195487'] 2023-03-07T10:23:13.899 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'tasks/{cls', 'validater/valgrind}'] 2023-03-07T10:23:13.899 INFO:scrape: 2023-03-07T10:23:13.900 INFO:scrape:Failure: Command failed on smithi133 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-03-07T10:23:13.900 INFO:scrape:1 jobs: ['7195464'] 2023-03-07T10:23:13.900 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-03-07T10:23:13.901 INFO:scrape: 2023-03-07T10:23:13.902 INFO:teuthology.results:Sending results to aemerson@redhat.com: Test Run: aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/ failed: 30 dead: 1 running: 1 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7195453] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195453/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b27449fe893b43aeb87b909329c6d14d Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195454] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:19:07 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195454/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195454/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da97ffd8d4104b03809fc29010398688 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195457] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:12:06 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195457/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfb235570d2541f380d44590f00e0462 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195458] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} ----------------------------------------------------------------- time: 00:08:44 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195458/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195458/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88828ed2b2764037aefce3691ca45f08 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195459] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195459/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c9b320c57a34646a1d78856c00c01d7 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195460] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:11:45 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195460/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b64f3188b2e492bbc225883729849da Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195462] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:08:45 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195462/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcc736783e3e4de3bf0aea1c9f5e3f5e S3ResponseError: 404 Not Found NoSuchKeymyfootx00000dcbf f9730c276729-0064066392-4234-default4234-default- default [7195463] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:11:24 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195463/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a39a116d7544e3d9c968d5f799e224a Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195464] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:09 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195464/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=224afad069474910b260fe1b6583baf2 Command failed on smithi133 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7195465] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195465/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3b58b2676d4491fb4e51c009eba66c7 Command failed (workunit test rgw/s3_bucket_quota.pl) on smithi017 with status 17: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_bucket_quota.pl' [7195466] rgw/tools/{centos_latest cluster tasks} ----------------------------------------------------------------- time: 00:18:50 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195466/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adf4744548b84b4cb0e1e4d2683e94f3 Command failed (workunit test rgw/test_rgw_orphan_list.sh) on smithi196 with status 12: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rgw/test_rgw_orphan_list.sh' [7195467] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:15:28 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195467/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ca4531cb60f4ed9880d0fb340a069a6 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195469] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195469/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=719bcabafbaf453b8af9a52150f8b0a7 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195470] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195470/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1067010a7d82491f959a37ac9ae3f4f9 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195471] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} ----------------------------------------------------------------- time: 00:13:10 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195471/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e54759d4178e4e85908a50844f02baa6 Command failed (workunit test rgw/s3_multipart_upload.pl) on smithi120 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4d310ef8b6c4505a84def3be96a8cdafa1dd0aa8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rgw/s3_multipart_upload.pl' [7195472] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:15:51 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195472/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195472/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d7aad9a26f44d4e93cc29506e0d7632 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195473] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest} ----------------------------------------------------------------- time: 00:08:41 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195473/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b897cc1363974e6c8c0dc5612c718f70 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195474] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:10:53 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195474/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b3db55b92694b839b968e2c2bedf5fe Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195475] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:43 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195475/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4eecd2155f344d849da4c9b13a03d84f S3ResponseError: 404 Not Found NoSuchKeymyfootx0000075ed a1543040a122-00640667df-4230-default4230-default- default [7195476] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest} ----------------------------------------------------------------- time: 00:08:55 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195476/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be32a9be1f794a14a53a7da06641db26 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195477] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195477/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195477/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce1f7d7b58964d1eba0f745fd57f6dbd Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195479] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:12:45 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195479/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195479/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b27c47e5e4594dc48a575abb525277d4 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195480] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} ----------------------------------------------------------------- time: 03:39:59 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195480/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b1c052621274b909703573cd94c8a4b Command failed on smithi053 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' [7195481] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:15:00 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195481/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8b95ea0fd4047b8b06aa0e12f089d22 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195482] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_user_quota ubuntu_latest} ----------------------------------------------------------------- time: 00:09:00 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195482/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86881f62a1074122892ac56108725b92 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195483] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:27 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195483/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2778001a70846429e43d803d1d369af Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' [7195484] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} ----------------------------------------------------------------- time: 00:08:55 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195484/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e91f1aeee8954953a905f152793b5175 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM' [7195485] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:36 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195485/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195485/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8db59acdfeb4bcbbd47a929cb65d393 S3ResponseError: 404 Not Found NoSuchKeymyfootx00000c09d 1a8011fc7f0f-0064066956-4229-default4229-default- default [7195488] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195488/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195488/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ffb95079ba549f0bc20cf69d1c7daa5 Command failed on smithi042 with status 100: 'sudo apt-get clean' [7195489] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:55 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195489/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58a55037ffe347ea96cb2b59559b7973 Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random' Dead ================================================================= [7195487] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195487/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195487/ Running ================================================================= [7195468] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest} info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195468/ Pass ================================================================= [7195455] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests} time: 00:18:46 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195455/ [7195456] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:18:51 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195456/ [7195461] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest} time: 00:14:34 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195461/ [7195478] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:16:47 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195478/ [7195486] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:31 info: http://pulpito.ceph.com/aemerson-2023-03-06_21:29:30-rgw-wip-remove-rgwsi_rados-distro-default-smithi/7195486/