2023-05-27T13:26:56.535 INFO:root:teuthology version: 0.0.1.dev111+gfa97ddd
2023-05-27T13:26:56.536 INFO:teuthology.results:seed: '4162'
2023-05-27T13:27:45.684 INFO:root:teuthology version: 0.0.1.dev111+gfa97ddd
2023-05-27T13:27:45.687 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-27T14:19:49.009 INFO:teuthology.results:Tests finished! gathering results...
2023-05-27T14:19:49.229 INFO:scrape:Found 6 jobs
2023-05-27T14:19:51.747 INFO:scrape:Found 2 distinct failure reasons
2023-05-27T14:19:51.748 INFO:scrape:Failure: Test failure: test_stale_write_caps (tasks.cephfs.test_client_recovery.TestClientRecovery)
2023-05-27T14:19:51.748 INFO:scrape:4 jobs: ['7288124', '7288122', '7288121', '7288125']
2023-05-27T14:19:51.749 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2023-05-27T14:19:51.749 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/client-recovery}']
2023-05-27T14:19:51.749 INFO:scrape:
2023-05-27T14:19:51.750 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
2023-05-27T14:19:51.750 INFO:scrape:2 jobs: ['7288120', '7288123']
2023-05-27T14:19:51.750 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2023-05-27T14:19:51.750 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2023-05-27T14:19:51.751 INFO:scrape:
2023-05-27T14:19:51.751 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7288120]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:27:57
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288120/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288120/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33e9f325dc204c09ab03c5d2968f9ea5

    Test failure: test_reconnect_after_blocklisted
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7288121]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:33:39
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288121/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288121/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e5dbcc9ecae9479bb243c2a2b1b4b8c5

    Test failure: test_stale_write_caps
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7288122]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:39:26
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288122/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288122/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e39d38f2343143809f27f94791e13687

    Test failure: test_stale_write_caps
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7288123]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:25:46
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288123/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288123/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1c294eafe6694ad584b79abf2946ca89

    Test failure: test_reconnect_after_blocklisted
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7288124]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:42:01
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288124/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288124/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba6b816d39474c4d9188b8358a3c2e26

    Test failure: test_stale_write_caps
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7288125]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:32:49
info:   http://pulpito.ceph.com/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288125/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-27_13:26:49-fs:functional-main-distro-default-smithi/7288125/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a23db5e737848ccbf33a97c13c9c5c4

    Test failure: test_stale_write_caps
    (tasks.cephfs.test_client_recovery.TestClientRecovery)