Found 39 jobs Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-16_13:35:39-fs:functional-wip-vshankar-testing-20240216.103400-testing-default-smithi/7562361/teuthology.log Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-16_13:35:39-fs:functional-wip-vshankar-testing-20240216.103400-testing-default-smithi/7562361/teuthology.log Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-16_13:35:39-fs:functional-wip-vshankar-testing-20240216.103400-testing-default-smithi/7562361/teuthology.log Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-16_13:35:39-fs:functional-wip-vshankar-testing-20240216.103400-testing-default-smithi/7562378/teuthology.log Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-16_13:35:39-fs:functional-wip-vshankar-testing-20240216.103400-testing-default-smithi/7562378/teuthology.log Found 8 distinct failure reasons Crash: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) ceph version 19.0.0-1514-gc2bc9db4 (c2bc9db4cb03566e1f716c0b72097e2a8afb77b8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff221c54db0] 2: (ceph::common::PerfCounters::set(int, unsigned long)+0x16) [0x7ff2226d5486] 3: (MetricAggregator::refresh_metrics_for_rank(entity_inst_t const&, int, Metrics const&)+0x1c7) [0x55fdff337e17] 4: (MetricAggregator::handle_mds_metrics(boost::intrusive_ptr const&)+0xef) [0x55fdff33c56f] 5: (MetricAggregator::ms_dispatch2(boost::intrusive_ptr const&)+0x9d) [0x55fdff3300ed] 6: (MetricAggregator::ms_fast_dispatch2(boost::intrusive_ptr const&)+0xe) [0x55fdff32c99e] 7: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x1b8) [0x7ff222775d38] 8: (DispatchQueue::run_local_delivery()+0x13c) [0x7ff222775edc] 9: /usr/lib64/ceph/libceph-common.so.2(+0x408ac1) [0x7ff222808ac1] 10: /lib64/libc.so.6(+0x9f802) [0x7ff221c9f802] 11: /lib64/libc.so.6(+0x3f450) [0x7ff221c3f450] 6 jobs: ['7562368', '7562358', '7562364', '7562349', '7562353', '7562350'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/data-scan}', 'tasks/exports}', 'tasks/journal-repair}', 'tasks/multimds_misc}', 'tasks/sessionmap}', 'tasks/strays}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0192d80d4a] ceph version 19.0.0-1514-gc2bc9db4 (c2bc9db4cb03566e1f716c0b72097e2a8afb77b8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f0192454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f01924a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0192d80d4a] 6: /usr/lib64/ceph/libceph-common.so.2(+0x180eae) [0x7f0192d80eae] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x354) [0x55afbfc49884] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55afbf9da1b9] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55afbf9dc59b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55afbf9dcaac] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55afbf9c4075] 12: (DispatchQueue::entry()+0x53a) [0x7f0192f766da] 13: /usr/lib64/ceph/libceph-common.so.2(+0x408aa1) [0x7f0193008aa1] 14: /lib64/libc.so.6(+0x9f802) [0x7f019249f802] 15: /lib64/libc.so.6(+0x3f450) [0x7f019243f450] 1 jobs: ['7562367'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/snapshots}'] Failure: Test failure: test_idem_unaffected_root_squash (tasks.cephfs.test_admin.TestFsAuthorizeUpdate) 1 jobs: ['7562375'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/admin}'] Dead: b"teuthology.exceptions.CommandFailedError: Command failed on smithi104 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'" ceph version 19.0.0-1514-gc2bc9db4 (c2bc9db4cb03566e1f716c0b72097e2a8afb77b8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3a94042520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x556ef0fc9821] 6: (KernelDevice::_aio_thread()+0x9e2) [0x556ef19692f2] 7: ceph-osd(+0xd97bc1) [0x556ef1969bc1] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3a94094b43] 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f3a94126a00] 1 jobs: ['7562372'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] Failure: SELinux denials found on ubuntu@smithi098.front.sepia.ceph.com: ['type=AVC msg=audit(1708091566.467:202): avc: denied { checkpoint_restore } for pid=1190 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 1 jobs: ['7562346'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/client-readahead}'] Dead: b'2024-02-17T01:53:01.882 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7562357'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/metrics}'] Failure: 1 jobs: ['7562379'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/backtrace}'] Dead: None 2 jobs: ['7562361', '7562378']