2020-12-02T02:03:02.418 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-12-02T02:03:02.420 INFO:teuthology.results:subset: '0/30000' 2020-12-02T02:03:02.420 INFO:teuthology.results:seed: '8631' 2020-12-02T02:20:16.991 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-12-02T02:20:17.034 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-02T03:24:18.304 INFO:teuthology.results:Tests finished! gathering results... 2020-12-02T03:24:24.636 INFO:scrape:Found 45 jobs 2020-12-02T03:24:25.411 INFO:scrape:Found 2 distinct failure reasons 2020-12-02T03:24:25.412 INFO:scrape:Failure: "2020-12-02T02:20:35.623038+0000 mon.a (mon.0) 192 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2020-12-02T03:24:25.413 INFO:scrape:1 jobs: ['5672052'] 2020-12-02T03:24:25.413 INFO:scrape:suites: ['frontend/civetweb', 'objectstore/filestore-xfs', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_multipart_upload}'] 2020-12-02T03:24:25.414 INFO:scrape: 2020-12-02T03:24:25.414 INFO:scrape:Failure: Command failed on smithi099 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds'" 2020-12-02T03:24:25.415 INFO:scrape:1 jobs: ['5672046'] 2020-12-02T03:24:25.415 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2020-12-02T03:24:25.415 INFO:scrape: 2020-12-02T03:24:25.417 INFO:teuthology.results:Sending results to aemerson@redhat.com: Test Run: aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 43 Fail ================================================================= [5672046] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:17:34 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672046/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672046/ sentry: https://sentry.ceph.com/organizations/ceph/?query=729dc87c4cee4dbb9b10d4e5fae1d048 Command failed on smithi099 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds'" [5672052] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_multipart_upload} ----------------------------------------------------------------- time: 00:06:45 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672052/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672052/ "2020-12-02T02:20:35.623038+0000 mon.a (mon.0) 192 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Pass ================================================================= [5672040] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests}} time: 00:18:35 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672040/ [5672041] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:13:51 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672041/ [5672042] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_bucket_quota} time: 00:09:13 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672042/ [5672043] rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} time: 00:38:46 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672043/ [5672044] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:10:35 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672044/ [5672045] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:27:15 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672045/ [5672047] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:15:03 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672047/ [5672048] rgw/tools/{centos_latest cluster tasks} time: 00:18:49 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672048/ [5672049] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:39:15 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672049/ [5672050] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:57 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672050/ [5672051] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:10:02 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672051/ [5672053] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:12:07 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672053/ [5672054] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} time: 00:25:48 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672054/ [5672055] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests}} time: 00:11:20 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672055/ [5672056] rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:24:08 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672056/ [5672057] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:24:53 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672057/ [5672058] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:04:32 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672058/ [5672059] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_ragweed} time: 00:07:05 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672059/ [5672060] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:12:20 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672060/ [5672061] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:25:26 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672061/ [5672062] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:13:11 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672062/ [5672063] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests}} time: 00:12:05 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672063/ [5672064] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:12:15 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672064/ [5672065] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_s3tests} time: 00:17:19 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672065/ [5672066] rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} time: 00:40:57 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672066/ [5672067] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:12:03 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672067/ [5672068] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:25:10 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672068/ [5672069] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:17:37 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672069/ [5672070] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:11:21 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672070/ [5672071] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:43:55 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672071/ [5672072] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:36 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672072/ [5672073] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:11:06 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672073/ [5672074] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_user_quota} time: 00:07:47 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672074/ [5672075] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:12:17 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672075/ [5672076] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} time: 00:25:22 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672076/ [5672077] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests}} time: 00:11:48 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672077/ [5672078] rgw/multisite/{clusters frontend/beast omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:24:00 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672078/ [5672079] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:20:46 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672079/ [5672080] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:45:33 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672080/ [5672081] rgw/multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_bucket_quota} time: 00:08:07 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672081/ [5672082] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:12:05 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672082/ [5672083] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:26:04 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672083/ [5672084] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:11:51 info: http://pulpito.ceph.com/aemerson-2020-12-01_16:16:20-rgw-wip-easter-bunny-distro-basic-smithi/5672084/