2022-07-19T19:34:23.637 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-19T19:34:23.639 INFO:teuthology.results:subset: '1/30000' 2022-07-19T19:34:23.639 INFO:teuthology.results:seed: '2845' 2022-07-19T19:59:09.070 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-19T19:59:09.095 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-19T22:31:25.172 INFO:teuthology.results:Tests finished! gathering results... 2022-07-19T22:31:29.217 INFO:scrape:Found 43 jobs 2022-07-19T22:31:29.530 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938544/teuthology.log matches 2022-07-19T22:31:29.531 WARNING:scrape:Misunderstood line: 2022-07-19T22:31:29.928 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938527/teuthology.log matches 2022-07-19T22:31:29.929 WARNING:scrape:Misunderstood line: 2022-07-19T22:31:31.326 INFO:scrape:Found 1 distinct failure reasons 2022-07-19T22:31:31.327 INFO:scrape:Valgrind: 2022-07-19T22:31:31.328 INFO:scrape:8 jobs: ['6938544', '6938527', '6938506', '6938507', '6938517', '6938538', '6938535', '6938513'] 2022-07-19T22:31:31.328 INFO:scrape:suites intersection: [] 2022-07-19T22:31:31.329 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'notify', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'omap_limits', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'reshard', 'rgw/cloud-transition/{cluster', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multisite/{clusters', 'rgw/tempest/{clusters/fixed-1', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_tempest', 'tasks/test_multi}', 'tasks/{cls', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}'] 2022-07-19T22:31:31.330 INFO:scrape: 2022-07-19T22:31:31.333 INFO:teuthology.results:Sending results to aemerson@redhat.com: Test Run: aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [6938506] rgw/cloud-transition/{cluster overrides supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} ----------------------------------------------------------------- time: 00:29:22 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938506/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d029d350b09441a79414cfa1c3122e1f Command failed (s3 tests against rgw) on smithi174 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a cloud_transition' [6938507] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:17:46 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938507/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07db1faff0b34415b8da45dc57a93853 Cannot create secret [6938513] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 01:24:30 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938513/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=650a93e7e6274edba22b6609b3a329cf rgw multisite test failures [6938517] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:27:32 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938517/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938517/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aab6e08f2055453eafaf560752ba6526 Command failed on smithi109 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6938527] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:51:25 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938527/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938527/ saw valgrind issues [6938535] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:32:30 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938535/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938535/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8dd5ee92b9c44a55b61047769768e5e9 rgw multisite test failures [6938538] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938538/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cabc03be0f4d4496930f2b176b3c725e reached maximum tries (90) after waiting for 540 seconds [6938544] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:41:05 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938544/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938544/ saw valgrind issues Pass ================================================================= [6938508] rgw/dbstore/{cluster overrides supported-random-distro$/{centos_8} tasks/rgw_s3tests} time: 00:15:43 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938508/ [6938509] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:17:23 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938509/ [6938510] rgw/iam/{cluster ignore-pg-availability objectstore pool-type rgw_frontend/beast supported_random_distro$/{rhel_8} tasks/{0-install iamtests}} time: 00:16:10 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938510/ [6938511] rgw/lifecycle/{cluster overrides supported-random-distro$/{centos_8} tasks/rgw_s3tests} time: 00:28:04 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938511/ [6938512] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:49 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938512/ [6938514] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:32:51 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938514/ [6938515] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:12:40 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938515/ [6938516] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:29:06 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938516/ [6938518] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:27 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938518/ [6938519] rgw/tools/{centos_latest cluster tasks} time: 02:45:56 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938519/ [6938520] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:41 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938520/ [6938521] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:20:25 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938521/ [6938522] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:50 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938522/ [6938523] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:17:43 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938523/ [6938524] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:12:04 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938524/ [6938525] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:15:10 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938525/ [6938526] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:20:20 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938526/ [6938528] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:17:01 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938528/ [6938529] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:16:20 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938529/ [6938530] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:13:07 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938530/ [6938531] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:15:32 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938531/ [6938532] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:15:16 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938532/ [6938533] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:16:39 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938533/ [6938534] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:09:39 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938534/ [6938536] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:14:32 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938536/ [6938537] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:39 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938537/ [6938539] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:15:24 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938539/ [6938540] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:15:27 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938540/ [6938541] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:10 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938541/ [6938542] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:14:34 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938542/ [6938543] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:12 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938543/ [6938545] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:26:21 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938545/ [6938546] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:12:09 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938546/ [6938547] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:12:43 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938547/ [6938548] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:01 info: http://pulpito.ceph.com/aemerson-2022-07-19_19:30:29-rgw-wip-chatty-bucket-sync-run-distro-default-smithi/6938548/