2023-07-09T20:21:15.135 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-09T20:21:15.136 INFO:teuthology.results:subset: '165/256'
2023-07-09T20:21:15.136 INFO:teuthology.results:seed: '4887'
2023-07-09T21:25:36.288 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-09T21:25:36.309 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-10T09:19:34.247 INFO:teuthology.results:Tests finished! gathering results...
2023-07-10T09:19:38.368 INFO:scrape:Found 43 jobs
2023-07-10T09:19:38.650 WARNING:scrape:Misunderstood line: 
2023-07-10T09:19:41.770 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/rishabh-2023-07-09_20:19:09-fs-wip-rishabh-2023Jul9-testing-default-smithi/7331687/teuthology.log
2023-07-10T09:20:04.644 INFO:scrape:Found 15 distinct failure reasons
2023-07-10T09:20:04.646 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-07-10T09:20:04.646 INFO:scrape:1 jobs: ['7331658']
2023-07-10T09:20:04.646 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-07-10T09:20:04.647 INFO:scrape:
2023-07-10T09:20:04.647 INFO:scrape:Failure: "2023-07-09T21:05:16.095940+0000 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 137242.920564 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-07-10T09:20:04.647 INFO:scrape:1 jobs: ['7331667']
2023-07-10T09:20:04.648 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-07-10T09:20:04.648 INFO:scrape:
2023-07-10T09:20:04.648 INFO:scrape:Failure: Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:35b2d86cfd512f15ace2077050d20c8669933212 pull'
2023-07-10T09:20:04.649 INFO:scrape:2 jobs: ['7331654', '7331668']
2023-07-10T09:20:04.649 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-07-10T09:20:04.649 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/mdtest}', 'tasks/{0-subvolume/{no-subvolume}']
2023-07-10T09:20:04.650 INFO:scrape:
2023-07-10T09:20:04.650 INFO:scrape:Failure: "2023-07-09T21:24:57.853825+0000 mon.a (mon.0) 373 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2023-07-10T09:20:04.650 INFO:scrape:1 jobs: ['7331677']
2023-07-10T09:20:04.651 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2023-07-10T09:20:04.651 INFO:scrape:
2023-07-10T09:20:04.651 INFO:scrape:Failure: Command failed on smithi132 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-07-10T09:20:04.652 INFO:scrape:3 jobs: ['7331659', '7331691', '7331682']
2023-07-10T09:20:04.652 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-07-10T09:20:04.652 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-07-10T09:20:04.653 INFO:scrape:
2023-07-10T09:20:04.653 INFO:scrape:Dead: None
2023-07-10T09:20:04.653 INFO:scrape:1 jobs: ['7331687']
2023-07-10T09:20:04.653 INFO:scrape:
2023-07-10T09:20:04.654 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b2d86cfd512f15ace2077050d20c8669933212 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-07-10T09:20:04.654 INFO:scrape:1 jobs: ['7331679']
2023-07-10T09:20:04.654 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2023-07-10T09:20:04.655 INFO:scrape:
2023-07-10T09:20:04.655 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-07-10T09:20:04.655 INFO:scrape:1 jobs: ['7331673']
2023-07-10T09:20:04.657 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-07-10T09:20:04.657 INFO:scrape:
2023-07-10T09:20:04.657 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-07-10T09:20:04.658 INFO:scrape:5 jobs: ['7331665', '7331678', '7331661', '7331655', '7331672']
2023-07-10T09:20:04.658 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2023-07-10T09:20:04.660 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2023-07-10T09:20:04.661 INFO:scrape:
2023-07-10T09:20:04.661 INFO:scrape:Dead: b'2023-07-10T09:13:16.346 DEBUG:teuthology.exit:Finished running handlers'
2023-07-10T09:20:04.661 INFO:scrape:ceph version 18.0.0-4759-g35b2d86c (35b2d86cfd512f15ace2077050d20c8669933212) reef (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fed37ccdce0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fed38f27e19]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2b6f85) [0x7fed38f27f85]
 6: (SimpleLock::put_xlock()+0x332) [0x55c990f83ad2]
 7: (Locker::xlock_finish(std::_Rb_tree_const_iterator<MutationImpl::LockOp> const&, MutationImpl*, bool*)+0x8c) [0x55c990f6bd5c]
 8: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x407) [0x55c990f74937]
 9: (Locker::drop_non_rdlocks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x5f) [0x55c990f791af]
 10: (Server::reply_client_request(boost::intrusive_ptr<MDRequestImpl>&, boost::intrusive_ptr<MClientReply> const&)+0x490) [0x55c990dc9040]
 11: (Server::respond_to_request(boost::intrusive_ptr<MDRequestImpl>&, int)+0x350) [0x55c990dc9eb0]
 12: (Server::_link_local_finish(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CInode*, unsigned long, unsigned long, bool)+0xf5) [0x55c990dcbdb5]
 13: (C_MDS_link_local_finish::finish(int)+0x35) [0x55c990e546b5]
 14: (MDSContext::complete(int)+0x5f) [0x55c9910b276f]
 15: (MDSIOContextBase::complete(int)+0x524) [0x55c9910b2ef4]
 16: (MDSLogContextBase::complete(int)+0x5b) [0x55c9910b32ab]
 17: (Finisher::finisher_thread_entry()+0x18d) [0x7fed38fcbacd]
 18: /lib64/libpthread.so.0(+0x81cf) [0x7fed37cc31cf]
 19: clone()
2023-07-10T09:20:04.662 INFO:scrape:1 jobs: ['7331683']
2023-07-10T09:20:04.662 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-07-10T09:20:04.662 INFO:scrape:
2023-07-10T09:20:04.663 INFO:scrape:Failure: Test failure: test_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)
2023-07-10T09:20:04.663 INFO:scrape:1 jobs: ['7331688']
2023-07-10T09:20:04.663 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}']
2023-07-10T09:20:04.664 INFO:scrape:
2023-07-10T09:20:04.664 INFO:scrape:Failure: Test failure: test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits)
2023-07-10T09:20:04.664 INFO:scrape:1 jobs: ['7331693']
2023-07-10T09:20:04.664 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2023-07-10T09:20:04.665 INFO:scrape:
2023-07-10T09:20:04.665 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi187 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b2d86cfd512f15ace2077050d20c8669933212 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-07-10T09:20:04.665 INFO:scrape:1 jobs: ['7331694']
2023-07-10T09:20:04.666 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-07-10T09:20:04.666 INFO:scrape:
2023-07-10T09:20:04.666 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b2d86cfd512f15ace2077050d20c8669933212 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'
2023-07-10T09:20:04.667 INFO:scrape:1 jobs: ['7331685']
2023-07-10T09:20:04.667 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-07-10T09:20:04.667 INFO:scrape:
2023-07-10T09:20:04.667 INFO:scrape:Failure: Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
2023-07-10T09:20:04.668 INFO:scrape:1 jobs: ['7331692']
2023-07-10T09:20:04.668 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}']
2023-07-10T09:20:04.668 INFO:scrape: