2021-04-08T16:59:03.369 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-08T16:59:03.374 INFO:teuthology.results:subset: '30/32' 2021-04-08T16:59:03.374 INFO:teuthology.results:seed: '3634' 2021-04-08T17:06:24.568 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-08T17:06:24.572 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-04-08T22:59:27.394 INFO:teuthology.results:Tests finished! gathering results... 2021-04-08T22:59:28.834 INFO:scrape:Found 13 jobs 2021-04-08T22:59:34.455 INFO:scrape:Found 7 distinct failure reasons 2021-04-08T22:59:34.456 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2021-04-08T22:59:34.457 INFO:scrape:1 jobs: ['6029206'] 2021-04-08T22:59:34.458 INFO:scrape:suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.458 INFO:scrape: 2021-04-08T22:59:34.459 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 2021-04-08T22:59:34.460 INFO:scrape:1 jobs: ['6029235'] 2021-04-08T22:59:34.460 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.461 INFO:scrape: 2021-04-08T22:59:34.461 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestSubCmdFsAuthorize) 2021-04-08T22:59:34.462 INFO:scrape:1 jobs: ['6029219'] 2021-04-08T22:59:34.462 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/admin}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.463 INFO:scrape: 2021-04-08T22:59:34.463 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f40190d964a62ca244e19afabfeef36902712876 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2021-04-08T22:59:34.464 INFO:scrape:ceph version 17.0.0-2954-gab29cb59 (ab29cb598ee15fcd52dd7f38c86ac0c13356d9be) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f09531bdb20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a0) [0x7f0954415496] 5: /usr/lib64/ceph/libceph-common.so.2(+0x27a650) [0x7f0954415650] 6: (MDSTableServer::handle_rollback(boost::intrusive_ptr const&)+0x3b4) [0x561bdfa65684] 7: (MDSTableServer::handle_request(boost::intrusive_ptr const&)+0x95) [0x561bdfa657b5] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xcc) [0x561bdf7a252c] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7bb) [0x561bdf7a568b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x55) [0x561bdf7a5c85] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x561bdf795c18] 12: (DispatchQueue::entry()+0x14ca) [0x7f09546736ea] 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f09547235e1] 14: (Thread::_entry_func(void*)+0xd) [0x7f0954511c5d] 15: /lib64/libpthread.so.0(+0x814a) [0x7f09531b314a] 16: clone() 2021-04-08T22:59:34.464 INFO:scrape:1 jobs: ['6029208'] 2021-04-08T22:59:34.465 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.465 INFO:scrape: 2021-04-08T22:59:34.465 INFO:scrape:Failure: Command failed on smithi072 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' 2021-04-08T22:59:34.466 INFO:scrape:1 jobs: ['6029229'] 2021-04-08T22:59:34.466 INFO:scrape:suites: ['2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.467 INFO:scrape: 2021-04-08T22:59:34.467 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi094 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f40190d964a62ca244e19afabfeef36902712876 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2021-04-08T22:59:34.468 INFO:scrape:1 jobs: ['6029232'] 2021-04-08T22:59:34.468 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.469 INFO:scrape: 2021-04-08T22:59:34.469 INFO:scrape:Failure: Test failure: test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes), test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes) 2021-04-08T22:59:34.469 INFO:scrape:1 jobs: ['6029226'] 2021-04-08T22:59:34.470 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-04-08T22:59:34.470 INFO:scrape: 2021-04-08T22:59:34.472 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6029206] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:11:12 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029206/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0448345f00ba4f1393bd51d891587df4 Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [6029208] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:36:22 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029208/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff0ca58ff42d4b768b40f33208073461 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f40190d964a62ca244e19afabfeef36902712876 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6029219] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} ----------------------------------------------------------------- time: 00:25:48 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029219/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07f4b83e5f174f508b3645f96fc86e60 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestSubCmdFsAuthorize) [6029226] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:45:23 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029226/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b996e30e725468e8cbc0c573706eb78 Test failure: test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes), test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes) [6029229] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:40:06 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029229/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0a9b9711bf1439abf349797835c2088 Command failed on smithi072 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' [6029232] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:50:08 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029232/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6cf9e3abf9948aabf79f8f982dc6301 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi094 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f40190d964a62ca244e19afabfeef36902712876 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6029235] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} ----------------------------------------------------------------- time: 00:16:10 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029235/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029235/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff9135fa7cfa4d3fbf22b3b2b9dba6d4 Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) Pass ================================================================= [6029210] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 05:49:00 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029210/ [6029213] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 01:02:49 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029213/ [6029216] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} time: 00:16:07 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029216/ [6029222] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:00:05 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029222/ [6029223] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 01:33:00 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029223/ [6029237] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 01:26:54 info: http://pulpito.ceph.com/pdonnell-2021-04-08_16:58:21-fs-wip-pdonnell-testing-20210408.142238-distro-basic-smithi/6029237/