2021-01-30T02:08:53.703 INFO:root:teuthology version: 1.0.0-f185165d 2021-01-30T02:08:53.704 INFO:teuthology.results:subset: '0/30000' 2021-01-30T02:08:53.705 INFO:teuthology.results:seed: '4209' 2021-01-30T02:44:42.152 INFO:root:teuthology version: 1.0.0-f185165d 2021-01-30T02:44:42.775 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-30T03:24:43.522 INFO:teuthology.results:Tests finished! gathering results... 2021-01-30T03:24:57.561 INFO:scrape:Found 45 jobs 2021-01-30T03:25:00.805 INFO:scrape:Found 4 distinct failure reasons 2021-01-30T03:25:00.806 INFO:scrape:Failure: Command failed on smithi139 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds'" 2021-01-30T03:25:00.807 INFO:scrape:1 jobs: ['5835250'] 2021-01-30T03:25:00.807 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-01-30T03:25:00.808 INFO:scrape: 2021-01-30T03:25:00.808 INFO:scrape:Failure: Command failed on smithi080 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2021-01-30T03:25:00.809 INFO:scrape:1 jobs: ['5835263'] 2021-01-30T03:25:00.809 INFO:scrape:suites: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/replicated', 'tasks/rgw_ragweed}'] 2021-01-30T03:25:00.809 INFO:scrape: 2021-01-30T03:25:00.810 INFO:scrape:Failure: "2021-01-30T02:40:14.442468+0000 mon.b (mon.0) 186 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2021-01-30T03:25:00.810 INFO:scrape:2 jobs: ['5835258', '5835279'] 2021-01-30T03:25:00.811 INFO:scrape:suites intersection: ['frontend/civetweb', 'overrides', 'rgw_pool_type/ec'] 2021-01-30T03:25:00.811 INFO:scrape:suites union: ['clusters/fixed-2', 'frontend/civetweb', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw/sts/{centos_latest', 'rgw_pool_type/ec', 'ststests', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/{0-install', 'webidentity}}'] 2021-01-30T03:25:00.811 INFO:scrape: 2021-01-30T03:25:00.812 INFO:scrape:Failure: rgw multisite pubsub, test failures 2021-01-30T03:25:00.812 INFO:scrape:1 jobs: ['5835270'] 2021-01-30T03:25:00.813 INFO:scrape:suites: ['frontend/civetweb', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2021-01-30T03:25:00.813 INFO:scrape: 2021-01-30T03:25:00.815 INFO:teuthology.results:Sending results to aemerson@redhat.com: Test Run: aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 40 Fail ================================================================= [5835250] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:17 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835250/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3177ff2fed1848b5980137851228fbb3 Command failed on smithi139 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds'" [5835258] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} ----------------------------------------------------------------- time: 00:24:50 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835258/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835258/ "2021-01-30T02:40:14.442468+0000 mon.b (mon.0) 186 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [5835263] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:11 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835263/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835263/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae04fdf874504421857249e52a28d2c5 Command failed on smithi080 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5835270] rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:32:33 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835270/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58f71ac7322149ba9cda61a3e5f876b6 rgw multisite pubsub, test failures [5835279] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:08:36 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835279/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835279/ "2021-01-30T02:56:10.640064+0000 mon.a (mon.0) 178 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log Pass ================================================================= [5835244] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests}} time: 00:18:48 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835244/ [5835245] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:19 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835245/ [5835246] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_bucket_quota} time: 00:09:01 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835246/ [5835247] rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} time: 00:29:51 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835247/ [5835248] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:12:04 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835248/ [5835249] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:25:14 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835249/ [5835251] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:11:57 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835251/ [5835252] rgw/tools/{centos_latest cluster tasks} time: 00:21:38 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835252/ [5835253] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:32:57 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835253/ [5835254] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:14:35 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835254/ [5835255] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:10:38 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835255/ [5835256] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_multipart_upload} time: 00:07:12 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835256/ [5835257] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:08:11 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835257/ [5835259] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests}} time: 00:11:29 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835259/ [5835260] rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:20:55 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835260/ [5835261] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:17:33 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835261/ [5835262] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:31:36 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835262/ [5835264] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:08:28 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835264/ [5835265] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:24:52 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835265/ [5835266] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:09:31 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835266/ [5835267] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests}} time: 00:11:47 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835267/ [5835268] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:12:22 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835268/ [5835269] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_s3tests} time: 00:11:55 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835269/ [5835271] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:11:12 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835271/ [5835272] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:24:31 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835272/ [5835273] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:19:34 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835273/ [5835274] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:12:03 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835274/ [5835275] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:30:23 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835275/ [5835276] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:15:38 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835276/ [5835277] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:11:10 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835277/ [5835278] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_user_quota} time: 00:07:21 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835278/ [5835280] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} time: 00:25:17 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835280/ [5835281] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests}} time: 00:11:47 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835281/ [5835282] rgw/multisite/{clusters frontend/beast omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:20:30 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835282/ [5835283] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:14:54 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835283/ [5835284] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:31:12 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835284/ [5835285] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_bucket_quota} time: 00:07:55 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835285/ [5835286] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:08:09 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835286/ [5835287] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:25:10 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835287/ [5835288] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:11:03 info: http://pulpito.ceph.com/aemerson-2021-01-28_15:57:30-rgw-wip-datalog-genbacking-distro-basic-smithi/5835288/