2023-08-08T12:36:12.018 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-08T12:36:12.019 INFO:teuthology.results:seed: '5270'
2023-08-08T12:50:24.307 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-08T12:50:24.311 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-08T13:47:28.772 INFO:teuthology.results:Tests finished! gathering results...
2023-08-08T13:47:29.549 INFO:scrape:Found 18 jobs
2023-08-08T13:47:29.551 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362794/teuthology.log
2023-08-08T13:47:33.891 INFO:scrape:Found 2 distinct failure reasons
2023-08-08T13:47:33.891 INFO:scrape:Dead: None
2023-08-08T13:47:33.891 INFO:scrape:1 jobs: ['7362794']
2023-08-08T13:47:33.892 INFO:scrape:
2023-08-08T13:47:33.892 INFO:scrape:Crash: "2023-08-08T13:05:07.385137+0000 mon.a (mon.0) 799 : cluster [WRN] Replacing daemon mds.c as rank 1 with standby daemon mds.d" in cluster log
2023-08-08T13:47:33.892 INFO:scrape:ceph version 18.0.0-5365-g9554f668 (9554f6689a5854fb8f551c2cc2b0a8297c9d74a1) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f532663c520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f5326d15857]
 6: /usr/lib/ceph/libceph-common.so.2(+0x1629b1) [0x7f5326d159b1]
 7: ceph-mds(+0x2c0b4b) [0x564840831b4b]
 8: (MDCache::split_dir(CDir*, int)+0x421) [0x56484083a531]
 9: ceph-mds(+0x1218ed) [0x5648406928ed]
 10: (CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x146) [0x7f5326dfaf36]
 11: /usr/lib/ceph/libceph-common.so.2(+0x2489b1) [0x7f5326dfb9b1]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f532668eb43]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5326720a00]
2023-08-08T13:47:33.893 INFO:scrape:6 jobs: ['7362789', '7362790', '7362793', '7362791', '7362792', '7362788']
2023-08-08T13:47:33.893 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}']
2023-08-08T13:47:33.893 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/fragment}']
2023-08-08T13:47:33.894 INFO:scrape:
2023-08-08T13:47:33.894 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/
failed:  6
dead:    1
running: 0
waiting: 0
queued:  0
passed:  11


Fail
=================================================================
[7362788]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:38:18
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362788/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362788/

    "2023-08-08T13:05:14.597105+0000 mon.a (mon.0) 710 : cluster [WRN]
    Replacing daemon mds.a as rank 1 with standby daemon mds.b" in cluster log

[7362789]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:38:12
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362789/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362789/

    "2023-08-08T13:05:07.385137+0000 mon.a (mon.0) 799 : cluster [WRN]
    Replacing daemon mds.c as rank 1 with standby daemon mds.d" in cluster log

[7362790]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:38:02
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362790/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362790/

    "2023-08-08T13:05:32.852248+0000 mon.a (mon.0) 804 : cluster [WRN]
    Replacing daemon mds.a as rank 1 with standby daemon mds.d" in cluster log

[7362791]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:43:57
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362791/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362791/

    "2023-08-08T13:11:29.545190+0000 mon.a (mon.0) 713 : cluster [WRN]
    Replacing daemon mds.b as rank 1 with standby daemon mds.c" in cluster log

[7362792]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:48:48
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362792/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362792/

    "2023-08-08T13:16:31.859505+0000 mon.a (mon.0) 721 : cluster [WRN]
    Replacing daemon mds.c as rank 1 with standby daemon mds.d" in cluster log

[7362793]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:48:54
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362793/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362793/

    "2023-08-08T13:17:57.804902+0000 mon.a (mon.0) 811 : cluster [WRN]
    Replacing daemon mds.d as rank 1 with standby daemon mds.a" in cluster log



Dead
=================================================================
[7362794]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362794/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362794/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Pass
=================================================================
[7362782] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:41:25
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362782/

[7362783] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:41:28
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362783/

[7362784] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:40:26
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362784/

[7362785] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:43:00
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362785/

[7362786] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:44:45
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362786/

[7362787] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:41:53
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362787/

[7362795] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:44:12
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362795/

[7362796] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:37:59
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362796/

[7362797] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:44:04
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362797/

[7362798] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:43:11
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362798/

[7362799] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:47:28
info:   http://pulpito.ceph.com/pdonnell-2023-08-08_12:34:14-fs-wip-batrick-testing-20230808.012441-distro-default-smithi/7362799/