Found 156 jobs Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2021-06-04T21:59:50.509 INFO:teuthology.orchestra.run.smithi081.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Found 10 distinct failure reasons Dead: b'2021-06-05T09:16:41.758 DEBUG:teuthology.exit:Finished running handlers' ceph version 16.2.4-305-g27fdd28e (27fdd28e981a347737df648ac2e2a273a515cd55) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f10dae363c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x7f10db0e365c] 5: (operator<<(std::ostream&, ClientMetricType const&)+0x163) [0x7f10db3862a3] 6: (MClientMetrics::print(std::ostream&) const+0x169) [0x7f10db3b52e9] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x788) [0x7f10db32a3b8] 8: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x2d) [0x7f10db32ab2d] 9: (ProtocolV1::handle_message_footer(char*, int)+0x11c7) [0x7f10db3fd537] 10: /usr/lib/ceph/libceph-common.so.2(+0x57a94a) [0x7f10db3f994a] 11: (AsyncConnection::process()+0x654) [0x7f10db3e7474] 12: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x595) [0x7f10db448695] 13: /usr/lib/ceph/libceph-common.so.2(+0x5d1d0a) [0x7f10db450d0a] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7f10dad10d84] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f10dae2a609] 16: clone() 22 jobs: ['6152438', '6152516', '6152524', '6152472', '6152602', '6152568', '6152626', '6152642', '6152591', '6152408', '6152638', '6152658', '6152444', '6152552', '6152670', '6152686', '6152450', '6152546', '6152624', '6152464', '6152560', '6152698'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'fs/multifs/{begin', 'fs/thrash/multifs/{begin', 'fs/volumes/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/centos_latest', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'session_timeout', 'tasks/alternate-pool}', 'tasks/cap-flush}', 'tasks/damage}', 'tasks/failover}', 'tasks/fragment}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mds-full}', 'tasks/mds_creation_retry}', 'tasks/openfiletable}', 'tasks/sessionmap}', 'tasks/strays}', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/misc}}', 'test/snapshot}}', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test suites/iozone.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27fdd28e981a347737df648ac2e2a273a515cd55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' ceph version 16.2.4-305-g27fdd28e (27fdd28e981a347737df648ac2e2a273a515cd55) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7eff3905f3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x7eff3930c65c] 5: (operator<<(std::ostream&, ClientMetricType const&)+0x163) [0x7eff395af2a3] 6: (MClientMetrics::print(std::ostream&) const+0x169) [0x7eff395de2e9] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x788) [0x7eff395533b8] 8: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x2d) [0x7eff39553b2d] 9: (ProtocolV1::handle_message_footer(char*, int)+0x11c7) [0x7eff39626537] 10: /usr/lib/ceph/libceph-common.so.2(+0x57a94a) [0x7eff3962294a] 11: (AsyncConnection::process()+0x654) [0x7eff39610474] 12: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x595) [0x7eff39671695] 13: /usr/lib/ceph/libceph-common.so.2(+0x5d1d0a) [0x7eff39679d0a] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7eff38f39d84] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7eff39053609] 16: clone() 12 jobs: ['6152406', '6152536', '6152478', '6152672', '6152430', '6152685', '6152468', '6152687', '6152456', '6152607', '6152578', '6152660'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/fsstress', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/fs/norstats}}', 'workunit/fs/test_o_trunc}}', 'workunit/kernel_untar_build}}', 'workunit/suites/iozone}}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi041 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27fdd28e981a347737df648ac2e2a273a515cd55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 1 jobs: ['6152532'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test suites/fsstress.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27fdd28e981a347737df648ac2e2a273a515cd55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' ceph version 16.2.4-305-g27fdd28e (27fdd28e981a347737df648ac2e2a273a515cd55) pacific (stable) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f67af588dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7f67b07dd3f5] 5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7f67b0a61d2e] 6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7f67b0a61f01] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x710) [0x7f67b0a15120] 8: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x32) [0x7f67b0a158e2] 9: (ProtocolV1::handle_message_footer(char*, int)+0x1500) [0x7f67b0ad3f40] 10: /usr/lib64/ceph/libceph-common.so.2(+0x56550d) [0x7f67b0acb50d] 11: (AsyncConnection::process()+0x789) [0x7f67b0ab9aa9] 12: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x7f67b0b13e27] 13: /usr/lib64/ceph/libceph-common.so.2(+0x5b433c) [0x7f67b0b1a33c] 14: /lib64/libstdc++.so.6(+0xc2b73) [0x7f67ae9c6b73] 15: /lib64/libpthread.so.0(+0x82de) [0x7f67af57e2de] 16: clone() 9 jobs: ['6152572', '6152558', '6152593', '6152699', '6152620', '6152440', '6152526', '6152636', '6152666'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/fsstress', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}}', 'workunit/suites/fsync-tester}}', 'workunit/suites/pjd}}'] Dead: b'2021-06-05T00:33:31.635 INFO:teuthology.orchestra.run.smithi002.stdout: 524300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 5 00:32 /home/ubuntu/cephtest/archive/syslog' ceph version 16.2.4-305-g27fdd28e (27fdd28e981a347737df648ac2e2a273a515cd55) pacific (stable) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f1c6ad06dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7f1c6bf5b3f5] 5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7f1c6c1dfd2e] 6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7f1c6c1dff01] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x710) [0x7f1c6c193120] 8: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x32) [0x7f1c6c1938e2] 9: (ProtocolV1::handle_message_footer(char*, int)+0x1500) [0x7f1c6c251f40] 10: /usr/lib64/ceph/libceph-common.so.2(+0x56550d) [0x7f1c6c24950d] 11: (AsyncConnection::process()+0x789) [0x7f1c6c237aa9] 12: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x7f1c6c291e27] 13: /usr/lib64/ceph/libceph-common.so.2(+0x5b433c) [0x7f1c6c29833c] 14: /lib64/libstdc++.so.6(+0xc2b73) [0x7f1c6a144b73] 15: /lib64/libpthread.so.0(+0x82de) [0x7f1c6acfc2de] 16: clone() 13 jobs: ['6152693', '6152428', '6152466', '6152585', '6152550', '6152697', '6152442', '6152644', '6152691', '6152682', '6152595', '6152616', '6152609'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'fs/multifs/{begin', 'fs/thrash/multifs/{begin', 'fs/volumes/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/centos_latest', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'session_timeout', 'tasks/admin}', 'tasks/backtrace}', 'tasks/client-recovery}', 'tasks/forward-scrub}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mds-full}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/scrub}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/clone}}', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2021-06-05T11:25:46.324 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6152652'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_21501 (tasks.cephfs.test_volume_client.TestVolumeClient) 1 jobs: ['6152514'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/volume-client}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['6152512'] suites: ['centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] Failure: Command failed on smithi205 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 1 jobs: ['6152544'] suites: ['1-client', '2-upgrade', '3-verify}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/volumes/import-legacy/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'ubuntu_18.04}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-06-04T22:53:54.121624+0000 mon.a (mon.0) 1213 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log 2 jobs: ['6152599', '6152548'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'fs/volumes/{begin', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/client-limits}', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']