2021-02-08T16:15:44.443 INFO:root:teuthology version: 1.1.0-17c09965 2021-02-08T16:15:44.443 INFO:teuthology.results:seed: '1518' 2021-02-08T17:14:43.657 INFO:root:teuthology version: 1.1.0-17c09965 2021-02-08T17:14:43.713 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-08T17:33:44.088 INFO:teuthology.results:Tests finished! gathering results... 2021-02-08T17:33:46.510 INFO:scrape:Found 30 jobs 2021-02-08T17:33:57.339 INFO:scrape:Found 2 distinct failure reasons 2021-02-08T17:33:57.339 INFO:scrape:Failure: Command failed on gibba001 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2021-02-08T17:33:57.339 INFO:scrape:6 jobs: ['5868059', '5868039', '5868044', '5868049', '5868054', '5868034'] 2021-02-08T17:33:57.339 INFO:scrape:suites intersection: ['frontend/civetweb', 'overrides', 'rgw:multifs/{clusters/fixed-2', 'tasks/rgw_ragweed}'] 2021-02-08T17:33:57.339 INFO:scrape:suites union: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'rgw:multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'tasks/rgw_ragweed}'] 2021-02-08T17:33:57.340 INFO:scrape: 2021-02-08T17:33:57.340 INFO:scrape:Failure: "2021-02-08T16:53:53.612235+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2021-02-08T17:33:57.340 INFO:scrape:1 jobs: ['5868042'] 2021-02-08T17:33:57.340 INFO:scrape:suites: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'overrides', 'rgw:multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_bucket_quota}'] 2021-02-08T17:33:57.340 INFO:scrape: 2021-02-08T17:33:57.341 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 23 Fail ================================================================= [5868034] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:07:17 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868034/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868034/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f064a0503c0468aac12b862a9bef245 Command failed on gibba008 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5868039] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:09 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868039/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868039/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bde8cb9036b4b8e9290bcc4d1a627b7 Command failed on gibba025 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5868042] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868042/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868042/ "2021-02-08T16:53:53.612235+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [5868044] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:44 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868044/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868044/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2ac2dac8be14ce1aed880895328d517 Command failed on gibba004 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5868049] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:28 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868049/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868049/ sentry: https://sentry.ceph.com/organizations/ceph/?query=481cba281bf44cbd97b027b536727da5 Command failed on gibba013 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5868054] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:51 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868054/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71e26c25f0fc4da390110f81b69e741f Command failed on gibba020 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [5868059] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_ragweed} ----------------------------------------------------------------- time: 00:06:00 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868059/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf0428166074475ba69dd89409069ea1 Command failed on gibba001 with status 1: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' Pass ================================================================= [5868032] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_bucket_quota} time: 00:09:35 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868032/ [5868033] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_multipart_upload} time: 00:07:10 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868033/ [5868035] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_s3tests} time: 00:14:08 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868035/ [5868036] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_user_quota} time: 00:08:31 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868036/ [5868037] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_bucket_quota} time: 00:08:48 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868037/ [5868038] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:07:51 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868038/ [5868040] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:14:40 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868040/ [5868041] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:07:26 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868041/ [5868043] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:06:58 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868043/ [5868045] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_s3tests} time: 00:13:34 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868045/ [5868046] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_user_quota} time: 00:08:27 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868046/ [5868047] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_bucket_quota} time: 00:08:49 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868047/ [5868048] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_multipart_upload} time: 00:07:56 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868048/ [5868050] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_s3tests} time: 00:15:21 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868050/ [5868051] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_user_quota} time: 00:07:23 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868051/ [5868052] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_bucket_quota} time: 00:09:25 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868052/ [5868053] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:06:39 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868053/ [5868055] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:13:27 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868055/ [5868056] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:07:56 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868056/ [5868057] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:08:57 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868057/ [5868058] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:07:39 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868058/ [5868060] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_s3tests} time: 00:14:44 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868060/ [5868061] rgw:multifs/{clusters/fixed-2 frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_user_quota} time: 00:07:15 info: http://pulpito.ceph.com/mkogan-2021-02-08_16:14:23-rgw:multifs-wip-rgw-d3n-cache-upstreaming_i38-distro-basic-gibba/5868061/