2022-02-19T06:41:49.701 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-19T06:41:49.702 INFO:teuthology.results:subset: '111/120000' 2022-02-19T06:41:49.703 INFO:teuthology.results:seed: '5234' 2022-02-19T07:02:47.285 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-19T07:02:47.343 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-19T09:38:59.463 INFO:teuthology.results:Tests finished! gathering results... 2022-02-19T09:39:03.241 INFO:scrape:Found 40 jobs 2022-02-19T09:39:04.799 INFO:scrape:Found 4 distinct failure reasons 2022-02-19T09:39:04.800 INFO:scrape:Failure: rgw multisite test failures 2022-02-19T09:39:04.800 INFO:scrape:2 jobs: ['6694860', '6694838'] 2022-02-19T09:39:04.801 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2022-02-19T09:39:04.801 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2022-02-19T09:39:04.801 INFO:scrape: 2022-02-19T09:39:04.802 INFO:scrape:Failure: Command failed on smithi112 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-02-19T09:39:04.802 INFO:scrape:1 jobs: ['6694842'] 2022-02-19T09:39:04.803 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-02-19T09:39:04.803 INFO:scrape: 2022-02-19T09:39:04.804 INFO:scrape:Failure: {'smithi022.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}, 'smithi006.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}} 2022-02-19T09:39:04.804 INFO:scrape:1 jobs: ['6694849'] 2022-02-19T09:39:04.805 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/filestore-xfs', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_ragweed', 'ubuntu_latest}'] 2022-02-19T09:39:04.805 INFO:scrape: 2022-02-19T09:39:04.806 INFO:scrape:Failure: Cannot create secret 2022-02-19T09:39:04.806 INFO:scrape:1 jobs: ['6694834'] 2022-02-19T09:39:04.806 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{rhel_8}}'] 2022-02-19T09:39:04.807 INFO:scrape: 2022-02-19T09:39:04.809 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [6694834] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:30:52 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd703a176db84f52967e83dc5658c6e0 Cannot create secret [6694838] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 02:45:28 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694838/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03545a034dc24f208a44d7f67212ade4 rgw multisite test failures [6694842] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:57 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694842/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b665bfb5fd864a548d65ab91d2c17a0c Command failed on smithi112 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6694860] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 01:34:53 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694860/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d021ede8ecf04547a2728cc6df8f220b rgw multisite test failures Dead ================================================================= [6694849] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} ----------------------------------------------------------------- time: 00:03:34 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694849/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c23b63e080b94cf99f995f7bab982f1c {'smithi022.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}, 'smithi006.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}} Pass ================================================================= [6694835] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:15:17 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694835/ [6694836] rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:35:13 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694836/ [6694837] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:04 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694837/ [6694839] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:31:30 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694839/ [6694840] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:09:24 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694840/ [6694841] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:27:48 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694841/ [6694843] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:12:01 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694843/ [6694844] rgw/tools/{centos_latest cluster tasks} time: 02:43:54 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694844/ [6694845] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:14 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694845/ [6694846] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:12 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694846/ [6694847] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:57 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694847/ [6694848] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:12:04 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694848/ [6694850] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694850/ [6694851] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:35 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694851/ [6694852] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:43:31 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694852/ [6694853] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:24:47 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694853/ [6694854] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694854/ [6694855] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:09:16 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694855/ [6694856] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:10:05 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694856/ [6694857] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694857/ [6694858] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:19 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694858/ [6694859] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:22 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694859/ [6694861] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:22:41 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694861/ [6694862] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:49 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694862/ [6694863] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:25:36 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694863/ [6694864] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694864/ [6694865] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:13:30 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694865/ [6694866] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:08:30 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694866/ [6694867] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:23:46 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694867/ [6694868] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694868/ [6694869] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:41:08 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694869/ [6694870] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694870/ [6694871] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:28 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694871/ [6694872] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:11:38 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694872/ [6694873] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:12:02 info: http://pulpito.ceph.com/yuriw-2022-02-18_20:58:08-rgw-master-distro-default-smithi/6694873/