2023-11-08T19:55:41.760 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T19:55:41.761 INFO:teuthology.results:seed: '754' 2023-11-08T20:00:51.930 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T20:00:51.933 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-08T20:45:55.516 INFO:teuthology.results:Tests finished! gathering results... 2023-11-08T20:45:55.786 INFO:scrape:Found 6 jobs 2023-11-08T20:45:57.309 INFO:scrape:Found 2 distinct failure reasons 2023-11-08T20:45:57.310 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fa6dd728008] 2023-11-08T20:45:57.310 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa6dd062520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fa6dd728008] 6: /usr/lib/ceph/libceph-common.so.2(+0x170162) [0x7fa6dd728162] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x341) [0x55c0735a2c01] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55c07333b969] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x55c07333d9eb] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c07333debc] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55c0733287f5] 12: (DispatchQueue::entry()+0x5ea) [0x7fa6dd913f2a] 13: /usr/lib/ceph/libceph-common.so.2(+0x3ee7f1) [0x7fa6dd9a67f1] 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa6dd0b4b43] 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa6dd146a00] 2023-11-08T20:45:57.311 INFO:scrape:1 jobs: ['7452117'] 2023-11-08T20:45:57.311 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-08T20:45:57.311 INFO:scrape: 2023-11-08T20:45:57.312 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7febf3654008] 2023-11-08T20:45:57.312 INFO:scrape:ceph version 18.0.0-7035-g7532011a (7532011aba9c81bdc6f987da6b4c05fb8ea5f4d4) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7febf2f8e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7febf3654008] 6: /usr/lib/ceph/libceph-common.so.2(+0x170162) [0x7febf3654162] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x341) [0x556c3b2cbc01] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x556c3b064969] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x556c3b0669eb] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556c3b066ebc] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x556c3b0517f5] 12: (DispatchQueue::entry()+0x5ea) [0x7febf383ff2a] 13: /usr/lib/ceph/libceph-common.so.2(+0x3ee7f1) [0x7febf38d27f1] 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7febf2fe0b43] 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7febf3072a00] 2023-11-08T20:45:57.312 INFO:scrape:1 jobs: ['7452114'] 2023-11-08T20:45:57.313 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-11-08T20:45:57.313 INFO:scrape: 2023-11-08T20:45:57.314 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [7452114] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:33:55 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452114/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452114/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e98d778bad24226b61bc51319642440 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7452117] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:33:37 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452117/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452117/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b571ed9bd8554487985cc5b9e4b72122 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) Pass ================================================================= [7452112] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:36:14 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452112/ [7452113] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:36:19 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452113/ [7452115] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:38:47 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452115/ [7452116] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:35:42 info: https://pulpito.ceph.com/pdonnell-2023-11-08_19:54:04-fs:functional-main-distro-default-smithi/7452116/