2023-11-04T01:04:39.733 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T01:04:39.733 INFO:teuthology.results:seed: '6533' 2023-11-04T01:04:42.073 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T01:04:42.074 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-04T02:02:46.432 INFO:teuthology.results:Tests finished! gathering results... 2023-11-04T02:02:46.661 INFO:scrape:Found 6 jobs 2023-11-04T02:02:50.746 INFO:scrape:Found 6 distinct failure reasons 2023-11-04T02:02:50.747 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa7a1d6f009] 2023-11-04T02:02:50.747 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fa7a1454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fa7a14a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa7a1d6f009] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f16d) [0x7fa7a1d6f16d] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x354) [0x55ebb4dd7254] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55ebb4b6a0a9] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ebb4b6cb2b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ebb4b6d01c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ebb4b56325] 12: (DispatchQueue::entry()+0x53a) [0x7fa7a1f5ff9a] 13: /usr/lib64/ceph/libceph-common.so.2(+0x3f1741) [0x7fa7a1ff1741] 14: /lib64/libc.so.6(+0x9f802) [0x7fa7a149f802] 15: /lib64/libc.so.6(+0x3f450) [0x7fa7a143f450] 2023-11-04T02:02:50.747 INFO:scrape:1 jobs: ['7445931'] 2023-11-04T02:02:50.747 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-04T02:02:50.747 INFO:scrape: 2023-11-04T02:02:50.747 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f645b96f009] 2023-11-04T02:02:50.747 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f645b054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f645b0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f645b96f009] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f16d) [0x7f645b96f16d] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x354) [0x55e89dd68254] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55e89dafb0a9] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55e89dafdb2b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e89dafe01c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55e89dae7325] 12: (DispatchQueue::entry()+0x53a) [0x7f645bb5ff9a] 13: /usr/lib64/ceph/libceph-common.so.2(+0x3f1741) [0x7f645bbf1741] 14: /lib64/libc.so.6(+0x9f802) [0x7f645b09f802] 15: /lib64/libc.so.6(+0x3f450) [0x7f645b03f450] 2023-11-04T02:02:50.747 INFO:scrape:1 jobs: ['7445933'] 2023-11-04T02:02:50.747 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-04T02:02:50.748 INFO:scrape: 2023-11-04T02:02:50.748 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe9cf56f009] 2023-11-04T02:02:50.748 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fe9cec54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fe9ceca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe9cf56f009] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f16d) [0x7fe9cf56f16d] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x354) [0x55b84cb20254] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55b84c8b30a9] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55b84c8b5b2b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b84c8b601c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55b84c89f325] 12: (DispatchQueue::entry()+0x53a) [0x7fe9cf75ff9a] 13: /usr/lib64/ceph/libceph-common.so.2(+0x3f1741) [0x7fe9cf7f1741] 14: /lib64/libc.so.6(+0x9f802) [0x7fe9cec9f802] 15: /lib64/libc.so.6(+0x3f450) [0x7fe9cec3f450] 2023-11-04T02:02:50.748 INFO:scrape:1 jobs: ['7445928'] 2023-11-04T02:02:50.748 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-04T02:02:50.748 INFO:scrape: 2023-11-04T02:02:50.748 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fed1f308008] 2023-11-04T02:02:50.748 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fed1ec42520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fed1f308008] 6: /usr/lib/ceph/libceph-common.so.2(+0x170162) [0x7fed1f308162] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x341) [0x55ea34d81c01] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55ea34b1a969] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x55ea34b1c9eb] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea34b1cebc] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ea34b077f5] 12: (DispatchQueue::entry()+0x5ea) [0x7fed1f4f3f2a] 13: /usr/lib/ceph/libceph-common.so.2(+0x3ee7f1) [0x7fed1f5867f1] 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fed1ec94b43] 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fed1ed26a00] 2023-11-04T02:02:50.748 INFO:scrape:1 jobs: ['7445930'] 2023-11-04T02:02:50.749 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-04T02:02:50.749 INFO:scrape: 2023-11-04T02:02:50.749 INFO:scrape:Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e3c29bf5b] 2023-11-04T02:02:50.749 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0e3c29bf5b] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2ba121) [0x7f0e3c29c121] 3: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x3e0) [0x555ece262340] 4: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x555ecdf23099] 5: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x593) [0x555ecdf261c3] 6: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555ecdf267bc] 7: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555ecdf1056f] 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0e3c5239c8] 9: (DispatchQueue::entry()+0x51f) [0x7f0e3c520b4f] 10: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0e3c5ee351] 11: /lib64/libpthread.so.0(+0x81cf) [0x7f0e3b0341cf] 2023-11-04T02:02:50.749 INFO:scrape:1 jobs: ['7445932'] 2023-11-04T02:02:50.749 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snapshots}'] 2023-11-04T02:02:50.749 INFO:scrape: 2023-11-04T02:02:50.749 INFO:scrape:Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf341c3f5b] 2023-11-04T02:02:50.749 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf341c3f5b] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2ba121) [0x7faf341c4121] 3: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x3e0) [0x555d5b532340] 4: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x555d5b1f3099] 5: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x593) [0x555d5b1f61c3] 6: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555d5b1f67bc] 7: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555d5b1e056f] 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7faf3444b9c8] 9: (DispatchQueue::entry()+0x51f) [0x7faf34448b4f] 10: (DispatchQueue::DispatchThread::entry()+0x11) [0x7faf34516351] 11: /lib64/libpthread.so.0(+0x81cf) [0x7faf32f5c1cf] 2023-11-04T02:02:50.750 INFO:scrape:1 jobs: ['7445929'] 2023-11-04T02:02:50.750 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snapshots}'] 2023-11-04T02:02:50.750 INFO:scrape: 2023-11-04T02:02:50.750 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7445928] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:26:00 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445928/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47603584283e40618ff2c5f3871af6ea Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7445929] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:23:16 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445929/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8c40bd156474b6fb97ec7c967ff03b5 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7445930] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:36:05 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445930/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445930/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4988b0ade5694eadb6dda2bb4cb66498 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7445931] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:25:12 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445931/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445931/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92711a75a12a4f7ea34ea3554e3add06 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7445932] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:23:51 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445932/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e21799ede6484e3981bf14758be71298 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7445933] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:32:01 info: http://pulpito.ceph.com/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445933/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-03_18:13:51-fs:functional-main-distro-default-smithi/7445933/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7060c9e7d9894023ad12c6e11d4e0818 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)