2020-12-23T18:55:02.784 INFO:root:teuthology version: 1.0.0-8e49dda4
2020-12-23T18:55:02.786 INFO:teuthology.results:seed: '9530'
2020-12-23T19:01:56.720 INFO:root:teuthology version: 1.0.0-8e49dda4
2020-12-23T19:01:57.675 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-12-23T20:01:58.687 INFO:teuthology.results:Tests finished! gathering results...
2020-12-23T20:01:59.946 INFO:scrape:Found 5 jobs
2020-12-23T20:02:01.025 INFO:scrape:Found 2 distinct failure reasons
2020-12-23T20:02:01.026 INFO:scrape:Failure: "2020-12-23T19:18:07.098321+0000 mon.b (mon.0) 325 : cluster [WRN] Health check failed: Degraded data redundancy: 3/6 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2020-12-23T20:02:01.026 INFO:scrape:2 jobs: ['5734684', '5734686']
2020-12-23T20:02:01.027 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2020-12-23T20:02:01.027 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2020-12-23T20:02:01.028 INFO:scrape:
2020-12-23T20:02:01.028 INFO:scrape:Failure: Command failed on smithi191 with status 22: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create '' 8"
2020-12-23T20:02:01.028 INFO:scrape:3 jobs: ['5734685', '5734687', '5734688']
2020-12-23T20:02:01.029 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2020-12-23T20:02:01.029 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2020-12-23T20:02:01.030 INFO:scrape:
2020-12-23T20:02:01.030 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[5734684]  fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:08:13
info:   http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734684/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734684/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d1e6f424b6d46f7b096a2f6ecd2b5f0

    "2020-12-23T19:18:07.098321+0000 mon.b (mon.0) 325 : cluster [WRN] Health
    check failed: Degraded data redundancy: 3/6 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[5734685]  fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:10:49
info:   http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734685/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734685/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8886aff9b62349d58474fa29b6a4aaa4

    Command failed on smithi191 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd pool create '' 8"

[5734686]  fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:08:21
info:   http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734686/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734686/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a2959dd9de14bc095cca46ba33222c4

    "2020-12-23T19:26:16.231824+0000 mon.b (mon.0) 351 : cluster [WRN] Health
    check failed: Degraded data redundancy: 3/6 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[5734687]  fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:10:25
info:   http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734687/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734687/
sentry: https://sentry.ceph.com/organizations/ceph/?query=89a27b37199c4dacae1a04da76b02923

    Command failed on smithi162 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd pool create '' 8"

[5734688]  fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:08:20
info:   http://pulpito.ceph.com/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734688/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-12-23_18:54:04-fs:workload-wip-pdonnell-testing-20201223.043716-distro-basic-smithi/5734688/
sentry: https://sentry.ceph.com/organizations/ceph/?query=097f28bb22fc41fb8e1558fa6040873b

    Command failed on smithi081 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd pool create '' 8"