2023-09-05T22:24:58.818 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556
2023-09-05T22:24:58.828 INFO:teuthology.results:subset: '400/512'
2023-09-05T22:24:58.828 INFO:teuthology.results:seed: '5483'
2023-09-05T22:38:30.180 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556
2023-09-05T22:38:30.183 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-05T23:47:35.641 INFO:teuthology.results:Tests finished! gathering results...
2023-09-05T23:47:36.620 INFO:scrape:Found 14 jobs
2023-09-05T23:47:37.933 INFO:scrape:Found 4 distinct failure reasons
2023-09-05T23:47:37.934 INFO:scrape:Failure: "2023-09-05T22:53:34.773602+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-09-05T23:47:37.934 INFO:scrape:2 jobs: ['7389199', '7389194']
2023-09-05T23:47:37.935 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health']
2023-09-05T23:47:37.935 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}', 'tasks/mds-flush}']
2023-09-05T23:47:37.936 INFO:scrape:
2023-09-05T23:47:37.936 INFO:scrape:Failure: Host key for server 'smithi123.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIE0fc0C4ixMVmGg5cVqpFjwztgo4Awrx5Q5fFTgCWNBq', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC4TBTjQj+LDweAt2PuxJoiqrZNg2k8DWWIjk/rG5N12WiRjWgbx/nUdWDPOzrw5K90C8I2rX6c+8aqJpDzcTmU8Gz9p0HKjQKJ1TrLHkOaeAk5be+BsJX+a6XiBLwiqUWJ9o2x2FxmruSptLlKhVc0cgGS0stAvIafm9otCWXYMknPk9KQ+GcbVZefPMI/yRgdZSaCLtFJFVDCDNmGUrUkxAVIo+WE2BLTQBT77zbzIaeGVKl5eUuUqC9Kq5uVqnXeUK4vGdMQxZcjWey9ThUUPq7zYWmquTRNuP0HdXWS7lW61CS7AqDrko05njfZklgEkHQvpSz328YbVJKnIRVGx6Gdm8YCMObCjWLdODZYe7PkOsJMt+Ezoqz/1x2Bd00ruUW3QqpxCg7/CjGVLn+8ryNw3WVBPUnwBWy7a2xfefUzqrezyzBSrkSeywGPagBcUrrQPvBCCEWquD2aFyqjoBerecGDJUAABPGg9J/vjlt5I575T96ms3Gg5HItA4c='
2023-09-05T23:47:37.937 INFO:scrape:1 jobs: ['7389205']
2023-09-05T23:47:37.937 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-09-05T23:47:37.937 INFO:scrape:
2023-09-05T23:47:37.938 INFO:scrape:Failure: Host key for server 'smithi012.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIBuQV4BEhd/6Gp7R0StlXzyJMQByFM04BRA4Rzo7Aq0z', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr3yv4mSTqqyGmbqo2QcQN2VgExOmChi/5O/leYZbWzU9t1MJhQxZbwUl0cQUPmsRhyo9+F9AFYm/CByYZSCY1NPIHjtIvORypW5j3WfbUFL8SQAa3StPmrTm1ybWhPDZGAYj3n/oUs0hwkfFBiroxmm4pMmiU25zsV9c9ZH+8dilYrllyoKnOr/l9QZfylYuGH1GhTNOQ2ypH6Q+2MkQ/smceQY4odpqQdA/gNDS1lq2rupuKW0gZ/6HhX5ZeDLkQu4JuvpIY6w1hCGg+kr1vrGgSQ/VNPKV1G4h7xfa7Z6dsnFHNEq7f7WQmaxwYMSpw52JmwmVNCIJlrtn0n0216XqB4COJ1Y9lA8twQiD1RZvnRL4QvVeBIYdCGPOA+uw3KF+m+GLtCtgQ3oQu3f8bINPMONzcFmrfb9ZWbWMwgGQP8E/f/Gzp34bGmFb1WzCXopf/RAtyC6D7vc1oLgc7kgqY/PUuB8GRoM7VkMwYU7YQKRrCzI50z788VG8ZZ4M='
2023-09-05T23:47:37.938 INFO:scrape:1 jobs: ['7389202']
2023-09-05T23:47:37.939 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}']
2023-09-05T23:47:37.939 INFO:scrape:
2023-09-05T23:47:37.939 INFO:scrape:Failure: Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
2023-09-05T23:47:37.940 INFO:scrape:1 jobs: ['7389196']
2023-09-05T23:47:37.940 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2023-09-05T23:47:37.940 INFO:scrape:
2023-09-05T23:47:37.942 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  9


Fail
=================================================================
[7389194]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   01:08:09
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389194/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389194/

    "2023-09-05T22:43:34.389289+0000 mon.a (mon.0) 208 : cluster [WRN] Health
    check failed: 2 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7389196]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:14:51
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389196/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389196/
sentry: https://sentry.ceph.com/organizations/ceph/?query=969a76758bae4e03b4ea669f018cb931

    Test failure: test_client_blocklisted_oldest_tid
    (tasks.cephfs.test_client_limits.TestClientLimits)

[7389199]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:14:13
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389199/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389199/

    "2023-09-05T22:53:34.773602+0000 mon.a (mon.0) 132 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7389202]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389202/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389202/
sentry: https://sentry.ceph.com/organizations/ceph/?query=206956cc988148f59ee748b20baf5fb7

    Host key for server 'smithi012.front.sepia.ceph.com' does not match: got
    'AAAAC3NzaC1lZDI1NTE5AAAAIBuQV4BEhd/6Gp7R0StlXzyJMQByFM04BRA4Rzo7Aq0z',
    expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr3yv4mSTqqyGmbqo2QcQN2VgExOmChi/5O
    /leYZbWzU9t1MJhQxZbwUl0cQUPmsRhyo9+F9AFYm/CByYZSCY1NPIHjtIvORypW5j3WfbUFL8S
    QAa3StPmrTm1ybWhPDZGAYj3n/oUs0hwkfFBiroxmm4pMmiU25zsV9c9ZH+8dilYrllyoKnOr/l
    9QZfylYuGH1GhTNOQ2ypH6Q+2MkQ/smceQY4odpqQdA/gNDS1lq2rupuKW0gZ/6HhX5ZeDLkQu4
    JuvpIY6w1hCGg+kr1vrGgSQ/VNPKV1G4h7xfa7Z6dsnFHNEq7f7WQmaxwYMSpw52JmwmVNCIJlr
    tn0n0216XqB4COJ1Y9lA8twQiD1RZvnRL4QvVeBIYdCGPOA+uw3KF+m+GLtCtgQ3oQu3f8bINPM
    ONzcFmrfb9ZWbWMwgGQP8E/f/Gzp34bGmFb1WzCXopf/RAtyC6D7vc1oLgc7kgqY/PUuB8GRoM7
    VkMwYU7YQKRrCzI50z788VG8ZZ4M='

[7389205]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389205/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389205/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aae2670033b747248bf3215d335d7970

    Host key for server 'smithi123.front.sepia.ceph.com' does not match: got
    'AAAAC3NzaC1lZDI1NTE5AAAAIE0fc0C4ixMVmGg5cVqpFjwztgo4Awrx5Q5fFTgCWNBq',
    expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC4TBTjQj+LDweAt2PuxJoiqrZNg2k8DWWIj
    k/rG5N12WiRjWgbx/nUdWDPOzrw5K90C8I2rX6c+8aqJpDzcTmU8Gz9p0HKjQKJ1TrLHkOaeAk5
    be+BsJX+a6XiBLwiqUWJ9o2x2FxmruSptLlKhVc0cgGS0stAvIafm9otCWXYMknPk9KQ+GcbVZe
    fPMI/yRgdZSaCLtFJFVDCDNmGUrUkxAVIo+WE2BLTQBT77zbzIaeGVKl5eUuUqC9Kq5uVqnXeUK
    4vGdMQxZcjWey9ThUUPq7zYWmquTRNuP0HdXWS7lW61CS7AqDrko05njfZklgEkHQvpSz328YbV
    JKnIRVGx6Gdm8YCMObCjWLdODZYe7PkOsJMt+Ezoqz/1x2Bd00ruUW3QqpxCg7/CjGVLn+8ryNw
    3WVBPUnwBWy7a2xfefUzqrezyzBSrkSeywGPagBcUrrQPvBCCEWquD2aFyqjoBerecGDJUAABPG
    g9J/vjlt5I575T96ms3Gg5HItA4c='



Pass
=================================================================
[7389192] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:30:30
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389192/

[7389193] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:17:14
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389193/

[7389195] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:13:05
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389195/

[7389197] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:24:22
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389197/

[7389198] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:15:53
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389198/

[7389200] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:29:43
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389200/

[7389201] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:13:12
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389201/

[7389203] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:31
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389203/

[7389204] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:19:19
info:   http://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389204/