Found 292 jobs Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902032/remote/smithi109/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz Not-understood crash indication Found 17 distinct failure reasons Failure: SSH connection to smithi083 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 7 jobs: ['6902075', '6901971', '6902048', '6901951', '6902005', '6901998', '6902095'] suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'osd}', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/pjd}', 'wsync/{yes}}'] Failure: reached maximum tries (90) after waiting for 540 seconds 4 jobs: ['6901884', '6902034', '6902084', '6901894'] suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: "2022-07-04T09:25:46.841103+0000 mon.a (mon.0) 1371 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log ceph version 17.2.0-501-gffefd1c0 (ffefd1c03754977bf5b4bbc82ce524944f0e0c52) quincy (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f571b21bb20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f571c237c32] 5: /usr/lib64/ceph/libceph-common.so.2(+0x283df5) [0x7f571c237df5] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x12b0) [0x556fce1818d0] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xfed) [0x556fcdead11d] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7cb) [0x556fcdeafabb] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fcdeb00dc] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x108) [0x556fcde9efd8] 11: (DispatchQueue::entry()+0x14fa) [0x7f571c4be43a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f571c575581] 13: /lib64/libpthread.so.0(+0x814a) [0x7f571b21114a] 14: clone() 1 jobs: ['6902125'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2022-07-01T15:45:58.717 DEBUG:teuthology.exit:Finished running handlers' 15 jobs: ['6902032', '6901852', '6901937', '6902049', '6902001', '6902025', '6902013', '6901860', '6902057', '6901939', '6901980', '6902058', '6902102', '6902040', '6902067'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{pg-warn', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/data-scan}', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/fsstress}', 'wsync/{no}}', 'wsync/{yes}}'] Failure: {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 7 jobs: ['6901921', '6901880', '6902093', '6902100', '6902105', '6901832', '6902076'] suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'fs/32bits/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{whitelist_health', 'ranks/1', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/norstats}', 'workunit/suites/ffsb}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'wsync/{yes}}'] Failure: Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 6 jobs: ['6902059', '6901758', '6901917', '6902091', '6901789', '6901794'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '1-cephadm', '1-client', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-upgrade', '2-workunit/suites/iozone}}', '3-compat_client/no}}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{secure}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{multimds/yes', 'overrides/{pg-warn', 'pg-warn', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-octopus', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/test_o_trunc}', 'workunit/suites/fsync-tester}', 'wsync/{no}}', 'wsync/{yes}}'] Failure: "2022-07-04T09:30:22.414806+0000 mon.a (mon.0) 1521 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2 jobs: ['6902119', '6901788'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 1 jobs: ['6901994'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{whitelist_health', 'tasks/workunit/snaps}', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 1 jobs: ['6901965'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' ceph version 17.2.0-501-gffefd1c0 (ffefd1c03754977bf5b4bbc82ce524944f0e0c52) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f55bd41cce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f55bdfe9c32] 5: /usr/lib64/ceph/libceph-common.so.2(+0x283df5) [0x7f55bdfe9df5] 6: (std::_Sp_counted_ptr::_M_dispose()+0x1cd) [0x5643753890dd] 7: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x47) [0x5643753860d7] 8: (Client::handle_client_session(boost::intrusive_ptr const&)+0x237) [0x56437536d017] 9: (Client::ms_dispatch2(boost::intrusive_ptr const&)+0x22d) [0x56437537101d] 10: (DispatchQueue::entry()+0x14fa) [0x7f55be27043a] 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55be327581] 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55bd4121ca] 13: clone() 1 jobs: ['6901759'] suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh' 1 jobs: ['6901872'] suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: SSH connection to smithi107 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' 1 jobs: ['6902070'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{osd-asserts', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones), test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 1 jobs: ['6902103'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: error during scrub thrashing: Command failed on smithi070 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' 1 jobs: ['6901774'] suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{yes}}'] Failure: Command failed (workunit test suites/fsstress.sh) on smithi042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 1 jobs: ['6902085'] suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: {'smithi099.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi161.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi066.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 1 jobs: ['6902097'] suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/test_o_trunc}', 'wsync/{yes}}'] Failure: Test failure: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 1 jobs: ['6902107'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}']