Found 155 jobs Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Not-understood crash indication 2021-03-25T05:06:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command crashed with signal 9 Not-understood crash indication Found 36 distinct failure reasons Failure: Test failure: test_clone_subvolume_info (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 1 jobs: ['5995366'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi031 with status 1: 'mkdir -- /home/ubuntu/cephtest/mnt.0' 14 jobs: ['5995412', '5995348', '5995374', '5995418', '5995332', '5995337', '5995402', '5995297', '5995353', '5995301', '5995388', '5995382', '5995303', '5995273'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsync-tester}}'] Failure: Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2) 1 jobs: ['5995271'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/multimds_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test kernel_untar_build.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 1 jobs: ['5995308'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}}'] Failure: Command failed (workunit test fs/snaps/snap-rm-diff.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh' 2 jobs: ['5995405', '5995380'] suites intersection: ['clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag_enable', 'tasks/workunit/snaps}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/workunit/snaps}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/norstats/kernel_untar_tar.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/norstats/kernel_untar_tar.sh' 1 jobs: ['5995358'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/norstats}}'] Failure: Test failure: test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap) 1 jobs: ['5995305'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/sessionmap}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 1 jobs: ['5995331'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test suites/iozone.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' 1 jobs: ['5995251'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iozone}}'] Failure: Command failed (workunit test suites/dbench.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' 2 jobs: ['5995345', '5995287'] suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'k-testing}', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/centos_latest', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: 'NoneType' object has no attribute 'mon_manager' 4 jobs: ['5995363', '5995387', '5995255', '5995296'] suites intersection: ['1-client', '2-upgrade', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) 1 jobs: ['5995398'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/fragment}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_replay_create (tasks.cephfs.test_cap_flush.TestCapFlush) 1 jobs: ['5995364'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/cap-flush}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 1 jobs: ['5995378'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-03-25T05:14:45.545810+0000 mds.a (mds.0) 14 : cluster [WRN] Scrub error on inode 0x100000003b1 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/commands) see mds.a log and `damage ls` output for details" in cluster log 3 jobs: ['5995367', '5995413', '5995330'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/mds_creation_retry}', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}', 'workunit/suites/fsstress}}'] Failure: Command failed (workunit test suites/fsstress.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 3 jobs: ['5995283', '5995355', '5995414'] suites intersection: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'tasks/fsstress', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout', 'tasks/fsstress', 'validater/lockdep}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 1 jobs: ['5995335'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed on smithi042 with status 1: 'cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0' ceph version 16.1.0-1201-ga3a37f39 (a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a) pacific (rc) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f8b119b4dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7f8b12c08b59] 5: (operator<<(std::ostream&, ClientMetricType const&)+0xcb) [0x7f8b12e8b02b] 6: (MClientMetrics::print(std::ostream&) const+0x1e9) [0x7f8b12e8b249] 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr const&)+0x710) [0x7f8b12e3f410] 8: (DispatchQueue::entry()+0xdeb) [0x7f8b12e40e7b] 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b12eef901] 10: /lib64/libpthread.so.0(+0x82de) [0x7f8b119aa2de] 11: clone() 5 jobs: ['5995285', '5995391', '5995426', '5995365', '5995342'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_clients (tasks.cephfs.test_failover.TestMultiFilesystems) 1 jobs: ['5995404'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/test_o_trunc.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a3a37f398e9d4da4d532fb63c7eb9ff22bb0a35a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh' 1 jobs: ['5995399'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/test_o_trunc}}'] Failure: Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks), test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) 1 jobs: ['5995302'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_barrier (tasks.cephfs.test_full.TestClusterFull) 2 jobs: ['5995410', '5995253'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_add_data_pool_non_alphamueric_name_as_subdir (tasks.cephfs.test_admin.TestAdminCommands) 1 jobs: ['5995344'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/admin}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) 2 jobs: ['5995281', '5995425'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/openfiletable}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/openfiletable}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) 1 jobs: ['5995361'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/backtrace}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 1 jobs: ['5995389'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) 1 jobs: ['5995293'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/snapshot}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 1 jobs: ['5995392'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/forward-scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: reached maximum tries (150) after waiting for 900 seconds 1 jobs: ['5995424'] suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 1 jobs: ['5995383'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 1 jobs: ['5995349'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/alternate-pool}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 1 jobs: ['5995307'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'socket.timeout' 1 jobs: ['5995421'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/centos_latest', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_basic (tasks.cephfs.test_client_recovery.TestClientRecovery) 1 jobs: ['5995375'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/client-recovery}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_dir_deletion (tasks.cephfs.test_strays.TestStrays), test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 1 jobs: ['5995327'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2021-03-25T17:07:03.296 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5995371'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']