2022-12-15T09:36:40.950 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-15T09:36:40.950 INFO:teuthology.results:seed: '2828' 2022-12-15T09:40:45.765 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-15T09:40:45.768 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-15T10:18:03.755 INFO:teuthology.results:Tests finished! gathering results... 2022-12-15T10:18:04.703 INFO:scrape:Found 6 jobs 2022-12-15T10:18:05.493 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118679/teuthology.log 2022-12-15T10:18:06.551 INFO:scrape:Found 4 distinct failure reasons 2022-12-15T10:18:06.552 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-12-15T10:18:06.553 INFO:scrape:2 jobs: ['7118677', '7118681'] 2022-12-15T10:18:06.553 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2022-12-15T10:18:06.553 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2022-12-15T10:18:06.553 INFO:scrape: 2022-12-15T10:18:06.553 INFO:scrape:Failure: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-12-15T10:18:06.554 INFO:scrape:1 jobs: ['7118676'] 2022-12-15T10:18:06.554 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}'] 2022-12-15T10:18:06.554 INFO:scrape: 2022-12-15T10:18:06.554 INFO:scrape:Dead: None 2022-12-15T10:18:06.554 INFO:scrape:1 jobs: ['7118679'] 2022-12-15T10:18:06.555 INFO:scrape: 2022-12-15T10:18:06.555 INFO:scrape:Failure: Test failure: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-12-15T10:18:06.555 INFO:scrape:2 jobs: ['7118680', '7118678'] 2022-12-15T10:18:06.555 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2022-12-15T10:18:06.555 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2022-12-15T10:18:06.556 INFO:scrape: 2022-12-15T10:18:06.557 INFO:teuthology.results:Sending results to neesingh@redhat.com: Test Run: neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/ failed: 5 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7118676] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118676/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118676/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddf01da95e124ca6ac055e4e1cd5f787 Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7118677] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:12:34 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118677/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118677/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40531f8078ac413aa13c378416e12185 Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7118678] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:27:41 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118678/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118678/ sentry: https://sentry.ceph.com/organizations/ceph/?query=395f8c018922464eaeaf150b6f5db803 Test failure: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7118680] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:20:36 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118680/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118680/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a390f986001644e2a645227ae6af25b7 Test failure: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7118681] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:16:53 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118681/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118681/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c262ab7d1534f80ab614fc3f6e22812 Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Dead ================================================================= [7118679] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118679/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-12-15_09:31:15-fs-wip-neesingh-54557-distro-default-smithi/7118679/ Error reimaging machines: Failed to power on smithi133