2023-03-06T13:29:04.898 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T13:29:04.899 INFO:teuthology.results:subset: '309/1024' 2023-03-06T13:29:04.900 INFO:teuthology.results:seed: '7060' 2023-03-06T14:05:46.995 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T14:05:47.021 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-07T02:06:42.745 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-07T02:06:46.122 INFO:scrape:Found 50 jobs 2023-03-07T02:06:50.374 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195203/teuthology.log 2023-03-07T02:06:51.992 WARNING:scrape:Misunderstood line: 2023-03-07T02:07:27.718 INFO:scrape:Found 19 distinct failure reasons 2023-03-07T02:07:27.719 INFO:scrape:Dead: b'2023-03-07T01:37:28.284 DEBUG:teuthology.exit:Finished running handlers' 2023-03-07T02:07:27.720 INFO:scrape:21 jobs: ['7195182', '7195213', '7195175', '7195200', '7195216', '7195192', '7195188', '7195210', '7195205', '7195183', '7195189', '7195196', '7195206', '7195211', '7195209', '7195197', '7195180', '7195208', '7195174', '7195195', '7195191'] 2023-03-07T02:07:27.720 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-03-07T02:07:27.721 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1-node-1-mds-1-osd', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides', 'overrides/whitelist_health', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{centos_8}', 'tasks/mgr-osd-full}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/snap-schedule}', 'tasks/snapshots}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/misc}}', 'test/snapshot}}', 'thrashosds-health}', 'wsync/no}', 'wsync/yes}'] 2023-03-07T02:07:27.721 INFO:scrape: 2023-03-07T02:07:27.721 INFO:scrape:Crash: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones), test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-03-07T02:07:27.722 INFO:scrape:ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7461e12cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7f7463aa94db] 5: (CDentry::check_corruption(bool)+0x614) [0x55801092e794] 6: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x558010787da7] 7: (Locker::_do_cap_update(CInode*, Capability*, int, snapid_t, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool*)+0x14da) [0x5580108bf7da] 8: (Locker::handle_client_caps(boost::intrusive_ptr const&)+0x25e4) [0x5580108d5804] 9: (Locker::dispatch(boost::intrusive_ptr const&)+0x1ec) [0x5580108d6b2c] 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x5580106bfa63] 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5580106c007c] 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5580106a96af] 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7463d208d8] 14: (DispatchQueue::entry()+0x50f) [0x7f7463d1da7f] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7463de3171] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7461e081ca] 17: clone() 2023-03-07T02:07:27.722 INFO:scrape:3 jobs: ['7195198', '7195217', '7195176'] 2023-03-07T02:07:27.722 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides'] 2023-03-07T02:07:27.723 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}', 'test/misc}}', 'test/snapshot}}'] 2023-03-07T02:07:27.723 INFO:scrape: 2023-03-07T02:07:27.723 INFO:scrape:Failure: Test failure: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-03-07T02:07:27.723 INFO:scrape:1 jobs: ['7195220'] 2023-03-07T02:07:27.724 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/forward-scrub}'] 2023-03-07T02:07:27.724 INFO:scrape: 2023-03-07T02:07:27.724 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi099 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-03-07T02:07:27.725 INFO:scrape:2 jobs: ['7195202', '7195187'] 2023-03-07T02:07:27.725 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-03-07T02:07:27.725 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-03-07T02:07:27.726 INFO:scrape: 2023-03-07T02:07:27.726 INFO:scrape:Failure: Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups), test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2023-03-07T02:07:27.726 INFO:scrape:1 jobs: ['7195190'] 2023-03-07T02:07:27.726 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-03-07T02:07:27.727 INFO:scrape: 2023-03-07T02:07:27.727 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-07T02:07:27.727 INFO:scrape:1 jobs: ['7195193'] 2023-03-07T02:07:27.728 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-03-07T02:07:27.728 INFO:scrape: 2023-03-07T02:07:27.728 INFO:scrape:Dead: None 2023-03-07T02:07:27.729 INFO:scrape:2 jobs: ['7195203', '7195171'] 2023-03-07T02:07:27.729 INFO:scrape: 2023-03-07T02:07:27.729 INFO:scrape:Failure: Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan) 2023-03-07T02:07:27.729 INFO:scrape:1 jobs: ['7195173'] 2023-03-07T02:07:27.730 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}'] 2023-03-07T02:07:27.730 INFO:scrape: 2023-03-07T02:07:27.730 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/libcephfs/test.sh 2023-03-07T02:07:27.731 INFO:scrape:1 jobs: ['7195181'] 2023-03-07T02:07:27.731 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-03-07T02:07:27.731 INFO:scrape: 2023-03-07T02:07:27.732 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-07T02:07:27.732 INFO:scrape:1 jobs: ['7195215'] 2023-03-07T02:07:27.732 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-03-07T02:07:27.732 INFO:scrape: 2023-03-07T02:07:27.733 INFO:scrape:Failure: 2023-03-07T02:07:27.733 INFO:scrape:1 jobs: ['7195179'] 2023-03-07T02:07:27.733 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-03-07T02:07:27.734 INFO:scrape: 2023-03-07T02:07:27.734 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-03-07T02:07:27.734 INFO:scrape:1 jobs: ['7195185'] 2023-03-07T02:07:27.734 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-03-07T02:07:27.735 INFO:scrape: 2023-03-07T02:07:27.735 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-07T02:07:27.735 INFO:scrape:2 jobs: ['7195219', '7195199'] 2023-03-07T02:07:27.736 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-03-07T02:07:27.736 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-03-07T02:07:27.736 INFO:scrape: 2023-03-07T02:07:27.737 INFO:scrape:Crash: Test failure: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) 2023-03-07T02:07:27.737 INFO:scrape:ceph version 18.0.0-2739-g472aaef4 (472aaef4ec3b3a2c29f73ca67854785a910f1f9b) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7efd5ba12cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7efd5d6a94db] 5: (CDentry::check_corruption(bool)+0x614) [0x55a2c0ef1794] 6: ceph-mds(+0x44e48a) [0x55a2c0f0c48a] 7: (CDir::_omap_commit(int)+0x2da) [0x55a2c0f0cb6a] 8: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x216) [0x55a2c0f0df36] 9: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xb67) [0x55a2c10579f7] 10: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a2c0fe592a] 11: (MDLog::trim_all()+0x764) [0x55a2c0fe7c24] 12: (C_Flush_Journal::trim_mdlog()+0x154) [0x55a2c0cb2254] 13: (Context::complete(int)+0xd) [0x55a2c0c6e13d] 14: (MDSContext::complete(int)+0x5f) [0x55a2c0fce2ff] 15: (C_IO_Wrapper::finish(int)+0x12) [0x55a2c0fcf1c2] 16: (MDSContext::complete(int)+0x5f) [0x55a2c0fce2ff] 17: (MDSIOContextBase::complete(int)+0x524) [0x55a2c0fcea84] 18: (Finisher::finisher_thread_entry()+0x18d) [0x7efd5d74c33d] 19: /lib64/libpthread.so.0(+0x81ca) [0x7efd5ba081ca] 20: clone() 2023-03-07T02:07:27.737 INFO:scrape:2 jobs: ['7195212', '7195172'] 2023-03-07T02:07:27.737 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-03-07T02:07:27.738 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-03-07T02:07:27.738 INFO:scrape: 2023-03-07T02:07:27.738 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2023-03-07T02:07:27.739 INFO:scrape:1 jobs: ['7195214'] 2023-03-07T02:07:27.739 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-03-07T02:07:27.739 INFO:scrape: 2023-03-07T02:07:27.739 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-03-07T02:07:27.740 INFO:scrape:1 jobs: ['7195186'] 2023-03-07T02:07:27.740 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-03-07T02:07:27.740 INFO:scrape: 2023-03-07T02:07:27.741 INFO:scrape:Dead: b'2023-03-07T02:07:17.952 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-03-07T02:07:27.741 INFO:scrape:1 jobs: ['7195218'] 2023-03-07T02:07:27.741 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-03-07T02:07:27.742 INFO:scrape: 2023-03-07T02:07:27.742 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2023-03-07T02:07:27.742 INFO:scrape:1 jobs: ['7195207'] 2023-03-07T02:07:27.742 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{rhel_8}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-03-07T02:07:27.743 INFO:scrape: 2023-03-07T02:07:27.743 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/test_python.sh 2023-03-07T02:07:27.743 INFO:scrape:1 jobs: ['7195194'] 2023-03-07T02:07:27.743 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-03-07T02:07:27.744 INFO:scrape: 2023-03-07T02:07:27.747 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/ failed: 21 dead: 15 running: 9 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7195172] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:22:55 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195172/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bee6607ff2cc4809b0817adc03358551 Test failure: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) [7195173] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:16:30 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195173/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8fe73d00b7c45ae80910b270f23f226 Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan) [7195176] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:30:45 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d35ebf449b145bbb7f100d54d65f904 Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc), test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) [7195179] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:16:53 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c8ce09ae54346e0a1617ed10d49e47b [7195181] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 03:20:08 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195181/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb6a83e9413e48b495fca378137dd265 Command failed (workunit test libcephfs/test.sh) on smithi157 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7195185] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:18:15 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195185/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05af380baf3148e6acd47338279748f5 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7195186] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:31:31 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195186/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98c6b5ed1116494d8eb15092f707452f Command failed (workunit test suites/fsstress.sh) on smithi067 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7195187] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:36:11 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195187/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8dc95a906d2407c939e028807f8d998 Command failed (workunit test kernel_untar_build.sh) on smithi090 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7195190] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:41:38 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195190/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f62465e6ed7e4d169b50ae3581568a97 Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups), test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups) [7195193] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:45 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195193/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=773ac56a52404a01b847a545ca3861f5 Command failed (workunit test suites/pjd.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7195194] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 03:19:16 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195194/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=422da25fdb4744178bf0a0979772d257 Command failed (workunit test fs/test_python.sh) on smithi033 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7195198] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:28:15 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195198/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10ed70eeec11444d943adad04aec53e6 Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones), test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7195199] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:46:18 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195199/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195199/ error during scrub thrashing: rank damage found: {'backtrace'} [7195202] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:33:38 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195202/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a71c144fe3fc4e36881d370cbd5eb481 Command failed (workunit test kernel_untar_build.sh) on smithi099 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7195207] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:40:16 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195207/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195207/ reached maximum tries (50) after waiting for 300 seconds [7195212] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:22:53 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195212/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195212/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e0801e7962845fdadeee961e06629ca Test failure: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) [7195214] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:20:49 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195214/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4cf5ad66ced4efc92c29b6ccfd0d04a Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [7195215] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:01:27 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d9ddd7212624738b1dfb31cfa579d73 saw valgrind issues [7195217] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:20:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195217/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b33b21c7174546b894e9a66052d2627f Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots), test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) [7195219] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:27:27 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195219/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195219/ error during scrub thrashing: rank damage found: {'backtrace'} [7195220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:24:40 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195220/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5f1d51521fc462fa6a16ee435de0c59 Test failure: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) Dead ================================================================= [7195171] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195171/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195171/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7195174] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195174/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195174/ hit max job timeout [7195175] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195175/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195175/ hit max job timeout [7195180] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195180/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195180/ hit max job timeout [7195182] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195182/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195182/ hit max job timeout [7195183] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195183/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195183/ hit max job timeout [7195189] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195189/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195189/ hit max job timeout [7195191] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195191/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195191/ hit max job timeout [7195192] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195192/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195192/ hit max job timeout [7195195] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195195/ hit max job timeout [7195196] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195196/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195196/ hit max job timeout [7195197] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195197/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195197/ hit max job timeout [7195200] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195200/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195200/ hit max job timeout [7195203] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195203/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195203/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7195205] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195205/ hit max job timeout Running ================================================================= [7195188] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195188/ [7195206] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195206/ [7195208] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195208/ [7195209] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195209/ [7195210] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195210/ [7195211] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195211/ [7195213] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195213/ [7195216] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195216/ [7195218] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195218/ Pass ================================================================= [7195177] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:32 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195177/ [7195178] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195178/ [7195184] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:24:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195184/ [7195201] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:24:18 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195201/ [7195204] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} time: 02:15:21 info: http://pulpito.ceph.com/vshankar-2023-03-06_13:28:53-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7195204/