2023-06-24T05:56:06.668 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb
2023-06-24T05:56:06.670 INFO:teuthology.results:seed: '2632'
2023-06-24T06:03:36.199 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb
2023-06-24T06:03:36.212 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-06-24T06:31:38.487 INFO:teuthology.results:Tests finished! gathering results...
2023-06-24T06:31:38.925 INFO:scrape:Found 10 jobs
2023-06-24T06:31:39.986 INFO:scrape:Found 1 distinct failure reasons
2023-06-24T06:31:39.987 INFO:scrape:Failure: Test failure: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity)
2023-06-24T06:31:39.988 INFO:scrape:10 jobs: ['7315425', '7315431', '7315432', '7315427', '7315434', '7315426', '7315433', '7315429', '7315428', '7315430']
2023-06-24T06:31:39.989 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}']
2023-06-24T06:31:39.989 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}']
2023-06-24T06:31:39.990 INFO:scrape:
2023-06-24T06:31:39.991 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/
failed:  10
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7315425]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:12:44
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315425/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315425/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b447d918e5949b18b052d2dced5eb87

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315426]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:15:59
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315426/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e8bc4db15bc44f7faff7bed8f8bf5e20

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315427]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:25:24
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315427/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315427/
sentry: https://sentry.ceph.com/organizations/ceph/?query=23e76e7fbdc043008dc42c90260d66ce

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315428]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:14:25
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315428/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315428/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33b339b64c714f00ac0effababd93070

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315429]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:17:08
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315429/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315429/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d5748eb42c642c58da446e220904e20

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315430]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:23:11
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315430/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315430/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d83443ec5f54cc9b28eba2cf191f4f4

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315431]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:17:03
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315431/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315431/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eff02d6a40ce4283a143f3f80269b381

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315432]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:14:56
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315432/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315432/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4fa28f6ddafd4ab0ba3bb6e861cf07c1

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315433]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:16:18
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315433/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d6c8c8c2567048df8ffbf585b5e40f87

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7315434]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:21:40
info:   http://pulpito.ceph.com/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315434/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2023-06-24_05:53:36-fs-wip-mgolub-testing-distro-default-smithi/7315434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96882f42b9714d028e4fa1c0a129a718

    Test failure: test_join_fs_last_resort
    (tasks.cephfs.test_failover.TestClusterAffinity)