2022-06-02T08:50:01.610 INFO:root:teuthology version: 1.1.1.dev360+g53fc7fef 2022-06-02T08:50:01.613 INFO:teuthology.results:subset: '1566/4096' 2022-06-02T08:50:01.613 INFO:teuthology.results:seed: '2140' 2022-06-02T09:05:51.616 INFO:root:teuthology version: 1.1.1.dev360+g53fc7fef 2022-06-02T09:05:51.638 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-02T10:56:04.615 INFO:teuthology.results:Tests finished! gathering results... 2022-06-02T10:56:12.430 INFO:scrape:Found 33 jobs 2022-06-02T10:56:30.543 INFO:scrape:Found 12 distinct failure reasons 2022-06-02T10:56:30.547 INFO:scrape:Failure: Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) 2022-06-02T10:56:30.547 INFO:scrape:2 jobs: ['6860118', '6860144'] 2022-06-02T10:56:30.547 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2022-06-02T10:56:30.547 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2022-06-02T10:56:30.547 INFO:scrape: 2022-06-02T10:56:30.549 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) 2022-06-02T10:56:30.549 INFO:scrape:2 jobs: ['6860119', '6860147'] 2022-06-02T10:56:30.549 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-flush}'] 2022-06-02T10:56:30.549 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-flush}'] 2022-06-02T10:56:30.557 INFO:scrape: 2022-06-02T10:56:30.557 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2022-06-02T10:56:30.561 INFO:scrape:1 jobs: ['6860139'] 2022-06-02T10:56:30.566 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/damage}'] 2022-06-02T10:56:30.566 INFO:scrape: 2022-06-02T10:56:30.566 INFO:scrape:Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2022-06-02T10:56:30.567 INFO:scrape:1 jobs: ['6860128'] 2022-06-02T10:56:30.567 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/strays}'] 2022-06-02T10:56:30.567 INFO:scrape: 2022-06-02T10:56:30.567 INFO:scrape:Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) 2022-06-02T10:56:30.567 INFO:scrape:1 jobs: ['6860134'] 2022-06-02T10:56:30.568 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/backtrace}'] 2022-06-02T10:56:30.572 INFO:scrape: 2022-06-02T10:56:30.572 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-06-02T10:56:30.574 INFO:scrape:2 jobs: ['6860115', '6860142'] 2022-06-02T10:56:30.574 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2022-06-02T10:56:30.575 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}'] 2022-06-02T10:56:30.575 INFO:scrape: 2022-06-02T10:56:30.577 INFO:scrape:Failure: Test failure: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2) 2022-06-02T10:56:30.580 INFO:scrape:1 jobs: ['6860121'] 2022-06-02T10:56:30.580 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multimds_misc}'] 2022-06-02T10:56:30.580 INFO:scrape: 2022-06-02T10:56:30.580 INFO:scrape:Failure: Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2022-06-02T10:56:30.580 INFO:scrape:1 jobs: ['6860124'] 2022-06-02T10:56:30.585 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2022-06-02T10:56:30.588 INFO:scrape: 2022-06-02T10:56:30.589 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2022-06-02T10:56:30.589 INFO:scrape:1 jobs: ['6860141'] 2022-06-02T10:56:30.596 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2022-06-02T10:56:30.600 INFO:scrape: 2022-06-02T10:56:30.605 INFO:scrape:Failure: "2022-06-02T09:25:23.099585+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 3 pgs degraded (PG_DEGRADED)" in cluster log 2022-06-02T10:56:30.605 INFO:scrape:3 jobs: ['6860132', '6860125', '6860120'] 2022-06-02T10:56:30.606 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2022-06-02T10:56:30.606 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/snap-schedule}'] 2022-06-02T10:56:30.606 INFO:scrape: 2022-06-02T10:56:30.606 INFO:scrape:Failure: "2022-06-02T09:22:51.185755+0000 mds.e (mds.0) 1 : cluster [WRN] client.4710 isn't responding to mclientcaps(revoke), ino 0x10000005b5e pending pFc issued pFcb, sent 300.010217 seconds ago" in cluster log 2022-06-02T10:56:30.610 INFO:scrape:1 jobs: ['6860126'] 2022-06-02T10:56:30.615 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-06-02T10:56:30.615 INFO:scrape: 2022-06-02T10:56:30.615 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2022-06-02T10:56:30.615 INFO:scrape:1 jobs: ['6860130'] 2022-06-02T10:56:30.615 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}'] 2022-06-02T10:56:30.616 INFO:scrape: 2022-06-02T10:56:30.618 INFO:teuthology.results:Sending results to ridave@redhat.com: Test Run: rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/ failed: 17 dead: 0 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [6860115] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:11:22 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860115/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12d2bd3714be4b8b8f7f7c9b992b5476 Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) [6860118] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860118/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d24e5e5ab044226811ef3857331d390 Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) [6860119] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:10:53 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860119/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860119/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a810aa267c124318ac665381408d04eb Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) [6860120] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:25:01 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860120/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860120/ "2022-06-02T09:20:30.337297+0000 mon.a (mon.0) 1495 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6860121] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:15:48 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860121/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c04c27859b5a4f49b7774d9521b2dd8c Test failure: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2) [6860124] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:13:28 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860124/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2fa893c6f4b45a893b9c38155039eec Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) [6860125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:54:15 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860125/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860125/ "2022-06-02T09:19:37.733875+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6860126] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:55:49 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860126/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860126/ "2022-06-02T09:22:51.185755+0000 mds.e (mds.0) 1 : cluster [WRN] client.4710 isn't responding to mclientcaps(revoke), ino 0x10000005b5e pending pFc issued pFcb, sent 300.010217 seconds ago" in cluster log [6860128] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860128/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe3ce13ba8174fe990e0210076805da2 Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) [6860130] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:23:23 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860130/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860130/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f90270f0b5f34f2d8acc900e46a4b8bd Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [6860132] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:30:05 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860132/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860132/ "2022-06-02T09:25:23.099585+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Degraded data redundancy: 1/96 objects degraded (1.042%), 3 pgs degraded (PG_DEGRADED)" in cluster log [6860134] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} ----------------------------------------------------------------- time: 00:16:46 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860134/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860134/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee6ff239fa7432ea94f8872a284f244 Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) [6860139] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:13:11 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860139/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860139/ sentry: https://sentry.ceph.com/organizations/ceph/?query=337f98811fd44ed6a8a419e743e1dad7 Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [6860141] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:17:54 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860141/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860141/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3a2685ca2ab4f1a9a5a8cc7892ed1bf Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [6860142] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:12:19 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860142/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c21f04ec7d6746c994597dd418e09003 Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) [6860144] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:11:08 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860144/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860144/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64238141968944a19d99194c0a4cb75c Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) [6860147] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:12:36 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860147/ log: http://qa-proxy.ceph.com/teuthology/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860147/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7df13990cca5498e842a7cfd9fa5c3e7 Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) Pass ================================================================= [6860116] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:27 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860116/ [6860117] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:46:23 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860117/ [6860122] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:22 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860122/ [6860123] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:12:34 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860123/ [6860127] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:54 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860127/ [6860129] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:07 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860129/ [6860131] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:10 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860131/ [6860133] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx}} time: 00:34:26 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860133/ [6860135] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:19 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860135/ [6860136] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:29 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860136/ [6860137] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:20 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860137/ [6860138] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:35:57 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860138/ [6860140] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:40 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860140/ [6860143] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:31:03 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860143/ [6860145] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:16:24 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860145/ [6860146] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:48 info: http://pulpito.ceph.com/rishabh-2022-06-02_04:47:25-fs-wip-vshankar-testing-20220527-073645-distro-basic-smithi/6860146/ 2022-06-02T10:56:30.975 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_53fc7fef6ccaf21fabe4a23a38668920f3ae41b8/teuthology/results.py", line 34, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_53fc7fef6ccaf21fabe4a23a38668920f3ae41b8/teuthology/results.py", line 82, in results body=body, File "/home/teuthworker/src/git.ceph.com_git_teuthology_53fc7fef6ccaf21fabe4a23a38668920f3ae41b8/teuthology/results.py", line 96, in email_results smtp.sendmail(msg['From'], [msg['To']], msg.as_string()) File "/usr/lib/python3.6/smtplib.py", line 867, in sendmail raise SMTPSenderRefused(code, resp, from_addr) smtplib.SMTPSenderRefused: (452, b'4.3.1 Insufficient system storage', 'ceph-qa@ceph.com')