2022-12-02T05:27:09.045 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-02T05:27:09.047 INFO:teuthology.results:subset: '1/20' 2022-12-02T05:27:09.047 INFO:teuthology.results:seed: '831' 2022-12-02T06:12:57.250 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-02T06:12:57.284 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-02T08:43:09.970 INFO:teuthology.results:Tests finished! gathering results... 2022-12-02T08:43:13.654 INFO:scrape:Found 39 jobs 2022-12-02T08:43:15.979 INFO:scrape:Found 2 distinct failure reasons 2022-12-02T08:43:15.980 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi132 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' 2022-12-02T08:43:15.980 INFO:scrape:1 jobs: ['7097322'] 2022-12-02T08:43:15.981 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] 2022-12-02T08:43:15.981 INFO:scrape: 2022-12-02T08:43:15.982 INFO:scrape:Failure: rgw multisite test failures 2022-12-02T08:43:15.982 INFO:scrape:2 jobs: ['7097321', '7097342'] 2022-12-02T08:43:15.983 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2022-12-02T08:43:15.983 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-12-02T08:43:15.984 INFO:scrape: 2022-12-02T08:43:15.986 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [7097321] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 02:07:26 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097321/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ab86a66b1a34516855142ff27a98cb2 rgw multisite test failures [7097322] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 02:09:32 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097322/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6b8b82e2b0f4dcb828e798390a0b9ff Command failed (bucket notification tests against different endpoints) on smithi132 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/ rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' [7097342] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 01:39:11 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097342/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8541fdcbf2d3416b97e51e10d697fab5 rgw multisite test failures Pass ================================================================= [7097314] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:45 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097314/ [7097315] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} time: 00:32:25 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097315/ [7097316] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:23:30 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097316/ [7097317] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:19:28 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097317/ [7097318] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:02 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097318/ [7097319] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} time: 00:23:15 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097319/ [7097320] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:44 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097320/ [7097323] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest} time: 00:14:10 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097323/ [7097324] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:47 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097324/ [7097325] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:30:15 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097325/ [7097326] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} time: 00:21:05 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097326/ [7097327] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:23 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097327/ [7097328] rgw/tools/{centos_latest cluster tasks} time: 02:46:29 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097328/ [7097329] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:17:28 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097329/ [7097330] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:12 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097330/ [7097331] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:16:24 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097331/ [7097332] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:08:47 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097332/ [7097333] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:15:37 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097333/ [7097334] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:19:01 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097334/ [7097335] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:13:04 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097335/ [7097336] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:14:17 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097336/ [7097337] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:10:58 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097337/ [7097338] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:10 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097338/ [7097339] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:15:46 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097339/ [7097340] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:14:31 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097340/ [7097341] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:09:01 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097341/ [7097343] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:53 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097343/ [7097344] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:13:09 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097344/ [7097345] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:59 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097345/ [7097346] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:15:50 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097346/ [7097347] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:10:27 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097347/ [7097348] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:10:51 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097348/ [7097349] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:25 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097349/ [7097350] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:13:33 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097350/ [7097351] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:21 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097351/ [7097352] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:20 info: http://pulpito.ceph.com/ivancich-2022-11-29_20:35:09-rgw-wip-fix-old-shard-distro-default-smithi/7097352/