2023-09-06T23:54:26.630 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-06T23:54:26.631 INFO:teuthology.results:subset: '100/128' 2023-09-06T23:54:26.631 INFO:teuthology.results:seed: '9261' 2023-09-06T23:55:14.558 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-06T23:55:14.561 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-07T00:56:18.484 INFO:teuthology.results:Tests finished! gathering results... 2023-09-07T00:56:18.947 INFO:scrape:Found 15 jobs 2023-09-07T00:56:20.624 INFO:scrape:Found 4 distinct failure reasons 2023-09-07T00:56:20.625 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4ba90dd80b1aa5b1a40fc707872b1636e7d0c3e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh' 2023-09-07T00:56:20.625 INFO:scrape:1 jobs: ['7390482'] 2023-09-07T00:56:20.625 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-09-07T00:56:20.626 INFO:scrape: 2023-09-07T00:56:20.626 INFO:scrape:Failure: "2023-09-07T00:20:54.997751+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi016:0 (7275), after 301.959 seconds" in cluster log 2023-09-07T00:56:20.626 INFO:scrape:1 jobs: ['7390488'] 2023-09-07T00:56:20.627 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2023-09-07T00:56:20.627 INFO:scrape: 2023-09-07T00:56:20.628 INFO:scrape:Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20a2969cc5] 2023-09-07T00:56:20.628 INFO:scrape:ceph version 18.0.0-5849-g00f23e9a (00f23e9a033b0c92a258985da15db21d2979c6d2) reef (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20a2969cc5] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2ace8b) [0x7f20a2969e8b] 3: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x3d3) [0x55d482ca8263] 4: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x55d482969a19] 5: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x593) [0x55d48296cb43] 6: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d48296d13c] 7: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d4829571ef] 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20a2be9ca8] 9: (DispatchQueue::entry()+0x51f) [0x7f20a2be6e2f] 10: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20a2cb18f1] 11: /lib64/libpthread.so.0(+0x81ca) [0x7f20a170d1ca] 2023-09-07T00:56:20.628 INFO:scrape:1 jobs: ['7390480'] 2023-09-07T00:56:20.629 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-09-07T00:56:20.629 INFO:scrape: 2023-09-07T00:56:20.629 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-09-07T00:56:20.630 INFO:scrape:1 jobs: ['7390478'] 2023-09-07T00:56:20.630 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-09-07T00:56:20.631 INFO:scrape: 2023-09-07T00:56:20.632 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [7390478] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:13:59 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390478/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5281ec16f43c49429d57c0733ed683d9 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7390480] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:28:01 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390480/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=988cc27b2f3e4ea7a3ace42d118be221 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7390482] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:09:08 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390482/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc4e90d3fe604ccbb78a055a5b9056a6 Command failed (workunit test fs/quota/quota.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4ba90dd80b1aa5b1a40fc707872b1636e7d0c3e7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh' [7390488] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:18:55 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390488/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390488/ "2023-09-07T00:20:54.997751+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi016:0 (7275), after 301.959 seconds" in cluster log Pass ================================================================= [7390476] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:24:15 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390476/ [7390477] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:20:08 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390477/ [7390479] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:18:48 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390479/ [7390481] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:15:03 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390481/ [7390483] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:13:36 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390483/ [7390484] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:18:15 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390484/ [7390485] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:14:12 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390485/ [7390486] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:50:56 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390486/ [7390487] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:52:16 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390487/ [7390489] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:15:11 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390489/ [7390490] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:11:05 info: http://pulpito.ceph.com/pdonnell-2023-09-06_23:54:19-fs:functional-main-distro-default-smithi/7390490/