2022-12-15T04:28:54.662 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-15T04:28:54.663 INFO:teuthology.results:seed: '4242'
2022-12-15T04:36:18.311 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-15T04:36:18.318 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-15T05:17:26.556 INFO:teuthology.results:Tests finished! gathering results...
2022-12-15T05:17:29.374 INFO:scrape:Found 6 jobs
2022-12-15T05:17:31.278 INFO:scrape:Found 3 distinct failure reasons
2022-12-15T05:17:31.278 INFO:scrape:Failure: Test failure: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-12-15T05:17:31.279 INFO:scrape:3 jobs: ['7117646', '7117647', '7117651']
2022-12-15T05:17:31.279 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-12-15T05:17:31.279 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}']
2022-12-15T05:17:31.280 INFO:scrape:
2022-12-15T05:17:31.280 INFO:scrape:Failure: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-12-15T05:17:31.280 INFO:scrape:1 jobs: ['7117649']
2022-12-15T05:17:31.280 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}']
2022-12-15T05:17:31.281 INFO:scrape:
2022-12-15T05:17:31.281 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-12-15T05:17:31.281 INFO:scrape:2 jobs: ['7117650', '7117648']
2022-12-15T05:17:31.281 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-12-15T05:17:31.282 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-12-15T05:17:31.282 INFO:scrape:
2022-12-15T05:17:31.283 INFO:teuthology.results:Sending results to neesingh@redhat.com: Test Run: neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7117646]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:20:29
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117646/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117646/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8f8edd38c7846779b8eee0bee825e33

    Test failure: test_health_status_after_dirfrag_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7117647]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:20:22
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117647/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117647/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96f6b673bce9482998bc680b147abab8

    Test failure: test_health_status_after_dirfrag_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7117648]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:17:21
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117648/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117648/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8af5c69cf3d54f00959d95ff12b18453

    Test failure: test_backtrace_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7117649]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117649/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117649/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b6319337e9cf4ade93f97d5267c8e29b

    Test failure: test_health_status_after_backtrace_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7117650]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:12:29
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117650/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117650/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de88b736f86a4c9487d9b4db81fad95a

    Test failure: test_backtrace_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7117651]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:26:43
info:   http://pulpito.ceph.com/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117651/
log:    http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_04:24:19-fs-wip-neesingh-fix-54557-distro-default-smithi/7117651/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d967c95fb7a14e71950b74dd210152b0

    Test failure: test_health_status_after_dirfrag_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)