2023-09-20T10:46:21.470 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-20T10:46:21.480 INFO:teuthology.results:subset: '614/1024' 2023-09-20T10:46:21.480 INFO:teuthology.results:seed: '2186' 2023-09-21T00:53:46.637 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-21T00:53:46.668 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-21T12:46:47.644 INFO:teuthology.results:Tests finished! gathering results... 2023-09-21T12:47:10.574 INFO:scrape:Found 203 jobs 2023-09-21T12:47:12.359 WARNING:scrape:Misunderstood line: 2023-09-21T12:47:17.123 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399213/teuthology.log 2023-09-21T12:47:32.990 WARNING:scrape:Misunderstood line: 2023-09-21T12:47:51.146 INFO:scrape:Found 15 distinct failure reasons 2023-09-21T12:47:51.147 INFO:scrape:Dead: b'2023-09-21T10:55:12.801 DEBUG:teuthology.exit:Finished running handlers' 2023-09-21T12:47:51.148 INFO:scrape:7 jobs: ['7399232', '7399181', '7399289', '7399318', '7399280', '7399276', '7399192'] 2023-09-21T12:47:51.148 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-21T12:47:51.149 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/2', '1-tests/fscrypt-common}}', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/misc}}', '5-workunit/suites/blogbench}}', '6-verify}}', 'begin/{0-install', 'bluestore-bitmap', 'centos_8.stream_container_tools', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fail_fs/yes', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/xfstests-dev}', 'tasks/{0-client', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-21T12:47:51.149 INFO:scrape: 2023-09-21T12:47:51.150 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-09-21T12:47:51.150 INFO:scrape:1 jobs: ['7399148'] 2023-09-21T12:47:51.151 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'fs/nfs/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}'] 2023-09-21T12:47:51.151 INFO:scrape: 2023-09-21T12:47:51.151 INFO:scrape:Failure: "2023-09-21T00:16:39.413291+0000 mon.a (mon.0) 599 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-09-21T12:47:51.152 INFO:scrape:6 jobs: ['7399284', '7399223', '7399144', '7399229', '7399249', '7399216'] 2023-09-21T12:47:51.152 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-21T12:47:51.153 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{centos_latest}', 'tasks/failover}', 'tasks/mirror}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/misc}}'] 2023-09-21T12:47:51.153 INFO:scrape: 2023-09-21T12:47:51.154 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-21T12:47:51.154 INFO:scrape:1 jobs: ['7399179'] 2023-09-21T12:47:51.154 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-09-21T12:47:51.155 INFO:scrape: 2023-09-21T12:47:51.155 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-09-21T12:47:51.156 INFO:scrape:3 jobs: ['7399182', '7399163', '7399329'] 2023-09-21T12:47:51.156 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-09-21T12:47:51.156 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-09-21T12:47:51.157 INFO:scrape: 2023-09-21T12:47:51.157 INFO:scrape:Failure: Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9d2b618-581c-11ee-9ab8-7b867c8bd7da -- ceph mon dump -f json' 2023-09-21T12:47:51.158 INFO:scrape:1 jobs: ['7399341'] 2023-09-21T12:47:51.158 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-09-21T12:47:51.159 INFO:scrape: 2023-09-21T12:47:51.159 INFO:scrape:Failure: "2023-09-20T22:27:27.547385+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 140689.301606 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-09-21T12:47:51.159 INFO:scrape:1 jobs: ['7399207'] 2023-09-21T12:47:51.160 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-09-21T12:47:51.160 INFO:scrape: 2023-09-21T12:47:51.161 INFO:scrape:Dead: None 2023-09-21T12:47:51.161 INFO:scrape:6 jobs: ['7399213', '7399258', '7399193', '7399278', '7399203', '7399224'] 2023-09-21T12:47:51.161 INFO:scrape: 2023-09-21T12:47:51.162 INFO:scrape:Failure: Command failed on smithi018 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-09-21T12:47:51.162 INFO:scrape:6 jobs: ['7399267', '7399248', '7399201', '7399265', '7399313', '7399215'] 2023-09-21T12:47:51.163 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-09-21T12:47:51.163 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-ffsb}}', '2-logrotate}', '2-workunit/ffsb}}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'session_timeout', 'tasks/cephfs_misc_tests}', 'tasks/mdtest}', 'tasks/strays}', 'tasks/{0-client', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-21T12:47:51.164 INFO:scrape: 2023-09-21T12:47:51.164 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-09-21T12:47:51.164 INFO:scrape:ceph version 18.0.0-6213-gbf8a061f (bf8a061ff16ac1d393cf108f740d92dfa49f2e54) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fd4b662bce0] 2: (CInode::get_dirfrags() const+0x26) [0x55635adadcc6] 3: (MDCache::advance_stray()+0x1fa) [0x55635ad1f69a] 4: (MDLog::_start_new_segment(SegmentBoundary*)+0x465) [0x55635af62a75] 5: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0xba) [0x55635af62c0a] 6: (MDLog::submit_entry(LogEvent*, MDSLogContextBase*)+0xbf) [0x55635ac3adaf] 7: (MDCache::shutdown_pass()+0xe9f) [0x55635ad78eef] 8: (MDSRankDispatcher::tick()+0x300) [0x55635abe0540] 9: (Context::complete(int)+0xd) [0x55635abb604d] 10: (CommonSafeTimer::timer_thread()+0x181) [0x7fd4b797fb61] 11: (CommonSafeTimerThread::entry()+0x11) [0x7fd4b7980e71] 12: /lib64/libpthread.so.0(+0x81cf) [0x7fd4b66211cf] 2023-09-21T12:47:51.165 INFO:scrape:1 jobs: ['7399238'] 2023-09-21T12:47:51.165 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-21T12:47:51.166 INFO:scrape: 2023-09-21T12:47:51.166 INFO:scrape:Failure: Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) 2023-09-21T12:47:51.166 INFO:scrape:1 jobs: ['7399326'] 2023-09-21T12:47:51.167 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-09-21T12:47:51.167 INFO:scrape: 2023-09-21T12:47:51.168 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_clone_cancel_pending (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-09-21T12:47:51.168 INFO:scrape:2 jobs: ['7399307', '7399176'] 2023-09-21T12:47:51.168 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-09-21T12:47:51.169 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-09-21T12:47:51.169 INFO:scrape: 2023-09-21T12:47:51.170 INFO:scrape:Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) 2023-09-21T12:47:51.170 INFO:scrape:1 jobs: ['7399153'] 2023-09-21T12:47:51.171 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2023-09-21T12:47:51.171 INFO:scrape: 2023-09-21T12:47:51.171 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-09-21T12:47:51.172 INFO:scrape:1 jobs: ['7399214'] 2023-09-21T12:47:51.172 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-21T12:47:51.173 INFO:scrape: 2023-09-21T12:47:51.173 INFO:scrape:Failure: Command failed (workunit test fs/full/subvolume_clone.sh) on smithi045 with status 11: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_clone.sh' 2023-09-21T12:47:51.173 INFO:scrape:1 jobs: ['7399345'] 2023-09-21T12:47:51.174 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-09-21T12:47:51.174 INFO:scrape: 2023-09-21T12:47:51.188 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/ failed: 26 dead: 13 running: 0 waiting: 0 queued: 0 passed: 164 Fail ================================================================= [7399144] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:18:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399144/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399144/ "2023-09-20T11:28:53.123932+0000 mon.a (mon.0) 2178 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7399148] fs/nfs/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:19:11 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399148/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16a44114fa7047fc9ac4faad3171eed7 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7399153] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:25:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399153/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e11674aa7f3143af8845c8f8d5970c9a Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) [7399163] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:21:01 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399163/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399163/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7399176] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:38:17 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=034c67b0bc0245ff8807bddb3f7a124d Test failure: test_subvolume_snapshot_clone_cancel_pending (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7399179] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:30:50 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd40c7a93f9848f0ae89930f75d2d769 saw valgrind issues [7399182] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:55:11 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399182/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399182/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7399201] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}} ----------------------------------------------------------------- time: 01:52:47 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399201/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399201/ Command failed on smithi027 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7399207] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:27:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399207/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399207/ "2023-09-20T22:27:27.547385+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 140689.301606 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7399214] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:30:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399214/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd383a808d8c4508bd3ed4cd4e433071 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7399215] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:15 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc697ecc4b494269b137bc52463222f6 Command failed on smithi161 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7399216] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:18:28 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399216/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399216/ "2023-09-20T23:23:51.517871+0000 mon.a (mon.0) 6187 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [7399223] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:21:03 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399223/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399223/ "2023-09-20T23:01:02.881772+0000 mon.a (mon.0) 692 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY)" in cluster log [7399229] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:23:40 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399229/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399229/ "2023-09-20T23:14:19.287270+0000 mon.a (mon.0) 490 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [7399238] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:37:10 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399238/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67ec7cc097a8438494fa8f948a0cc3f7 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7399248] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 01:01:33 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399248/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399248/ Command failed on smithi044 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7399249] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:37:33 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399249/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399249/ "2023-09-20T23:42:07.680403+0000 mon.a (mon.0) 815 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [7399265] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:25:04 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399265/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399265/ Command failed on smithi053 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7399267] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:33:52 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399267/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399267/ Command failed on smithi018 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7399284] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 02:10:32 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399284/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399284/ "2023-09-21T00:16:39.413291+0000 mon.a (mon.0) 599 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7399307] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:40:24 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399307/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399307/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9754bee024014dbf998a77c2d524560c Test failure: test_subvolume_snapshot_clone_cancel_pending (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7399313] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:50 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399313/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b6a6ef99f7b456b81dba8c24a221b7f Command failed on smithi149 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7399326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:20:45 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399326/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9989f453f7f4c1c87a4019cccd2d359 Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) [7399329] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:47:48 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399329/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399329/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7399341] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:56 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399341/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e937f01cc26c4c329fe56de40b223253 Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9d2b618-581c-11ee-9ab8-7b867c8bd7da -- ceph mon dump -f json' [7399345] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:20:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399345/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86ce9dc5acdd444baf466555f9e2ec5a Command failed (workunit test fs/full/subvolume_clone.sh) on smithi045 with status 11: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf8a061ff16ac1d393cf108f740d92dfa49f2e54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/full/subvolume_clone.sh' Dead ================================================================= [7399181] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399181/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399181/ hit max job timeout [7399192] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399192/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399192/ hit max job timeout [7399193] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399193/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399193/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7399203] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399203/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399203/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7399213] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399213/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7399224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399224/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399224/ Error reimaging machines: Timeout opening channel. [7399232] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399232/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399232/ hit max job timeout [7399258] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399258/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399258/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7399276] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399276/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399276/ hit max job timeout [7399278] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399278/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399278/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7399280] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399280/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399280/ hit max job timeout [7399289] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399289/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399289/ hit max job timeout [7399318] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399318/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399318/ hit max job timeout Pass ================================================================= [7399143] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:18:32 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399143/ [7399145] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit} time: 00:34:31 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399145/ [7399146] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:25 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399146/ [7399147] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:28:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399147/ [7399149] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:04 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399149/ [7399150] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:49:36 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399150/ [7399151] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:14:09 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399151/ [7399152] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:21:44 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399152/ [7399154] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:18:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399154/ [7399155] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:36:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399155/ [7399156] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:01 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399156/ [7399157] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399157/ [7399158] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:18:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399158/ [7399159] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:28 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399159/ [7399160] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:15:20 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399160/ [7399161] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:17:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399161/ [7399162] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:20:26 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399162/ [7399164] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:22:48 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399164/ [7399165] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:12 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399165/ [7399166] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:38 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399166/ [7399167] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:15:47 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399167/ [7399168] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:55:31 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399168/ [7399169] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:20:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399169/ [7399170] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 01:02:34 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399170/ [7399171] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399171/ [7399172] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:50:26 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399172/ [7399173] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:20:54 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399173/ [7399174] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399174/ [7399175] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:15 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399175/ [7399177] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:25:42 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399177/ [7399178] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:36 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399178/ [7399180] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:50:15 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399180/ [7399183] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:44:03 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399183/ [7399184] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:16:49 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399184/ [7399185] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399185/ [7399186] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:16 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399186/ [7399187] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 01:01:01 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399187/ [7399188] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:22:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399188/ [7399189] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 01:10:07 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399189/ [7399190] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:51 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399190/ [7399191] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:36 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399191/ [7399194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:24:02 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399194/ [7399195] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:18 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399195/ [7399196] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:12 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399196/ [7399197] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:23:53 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399197/ [7399198] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399198/ [7399199] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399199/ [7399200] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:33 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399200/ [7399202] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:28 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399202/ [7399204] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:21:32 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399204/ [7399205] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:14:27 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399205/ [7399206] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:48 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399206/ [7399208] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:37 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399208/ [7399209] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:34 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399209/ [7399210] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:24:18 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399210/ [7399211] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:42:12 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399211/ [7399212] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:03 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399212/ [7399217] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:43:32 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399217/ [7399218] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:29 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399218/ [7399219] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:32:20 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399219/ [7399220] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:21:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399220/ [7399221] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:52 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399221/ [7399222] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:42 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399222/ [7399225] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:30 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399225/ [7399226] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399226/ [7399227] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:23:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399227/ [7399228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:49 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399228/ [7399230] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:22:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399230/ [7399231] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:50 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399231/ [7399233] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} time: 00:35:10 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399233/ [7399234] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:18:15 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399234/ [7399235] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:21:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399235/ [7399236] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:17 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399236/ [7399237] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:18:51 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399237/ [7399239] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:53:38 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399239/ [7399240] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:44:08 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399240/ [7399241] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:27:11 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399241/ [7399242] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:04 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399242/ [7399243] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:32 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399243/ [7399244] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399244/ [7399245] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:55 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399245/ [7399246] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:23:53 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399246/ [7399247] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:27:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399247/ [7399250] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399250/ [7399251] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:15:34 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399251/ [7399252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:18:36 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399252/ [7399253] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399253/ [7399254] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:40:48 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399254/ [7399255] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:54:52 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399255/ [7399256] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:41 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399256/ [7399257] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:35:05 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399257/ [7399259] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:55:49 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399259/ [7399260] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:24:57 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399260/ [7399261] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:14:29 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399261/ [7399262] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:02 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399262/ [7399263] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:31:07 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399263/ [7399264] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:14:55 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399264/ [7399266] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:41:29 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399266/ [7399268] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:21:41 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399268/ [7399269] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:35 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399269/ [7399270] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:25 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399270/ [7399271] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:37 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399271/ [7399272] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:23:17 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399272/ [7399273] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:13 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399273/ [7399274] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:22:46 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399274/ [7399275] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:37:22 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399275/ [7399277] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:15:16 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399277/ [7399279] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:10 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399279/ [7399281] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:28 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399281/ [7399282] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:14:07 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399282/ [7399283] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:37:06 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399283/ [7399285] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:11 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399285/ [7399286] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:37:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399286/ [7399287] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:48 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399287/ [7399288] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:56:31 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399288/ [7399290] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:35:05 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399290/ [7399291] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:17:54 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399291/ [7399292] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:21:14 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399292/ [7399293] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:17 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399293/ [7399294] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:21:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399294/ [7399295] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:27:55 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399295/ [7399296] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:22:11 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399296/ [7399297] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:34:28 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399297/ [7399298] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:26:40 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399298/ [7399299] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:23 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399299/ [7399300] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:16:56 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399300/ [7399301] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:13 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399301/ [7399302] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:40:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399302/ [7399303] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:56:00 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399303/ [7399304] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:33 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399304/ [7399305] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:15:13 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399305/ [7399306] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:49:36 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399306/ [7399308] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:40:18 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399308/ [7399309] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:19:24 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399309/ [7399310] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:39 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399310/ [7399311] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:16 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399311/ [7399312] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:20:14 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399312/ [7399314] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399314/ [7399315] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:59:55 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399315/ [7399316] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:38 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399316/ [7399317] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:23:07 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399317/ [7399319] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:45 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399319/ [7399320] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:32:55 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399320/ [7399321] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:23:49 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399321/ [7399322] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:17:13 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399322/ [7399323] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:31 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399323/ [7399324] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:12 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399324/ [7399325] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:14 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399325/ [7399327] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:53 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399327/ [7399328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:52 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399328/ [7399330] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:13 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399330/ [7399331] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:04 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399331/ [7399332] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:30:43 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399332/ [7399333] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} time: 00:57:19 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399333/ [7399334] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:58:51 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399334/ [7399335] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:12 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399335/ [7399336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:44:21 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399336/ [7399337] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:24 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399337/ [7399338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:30:57 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399338/ [7399339] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:23:54 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399339/ [7399340] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:48:57 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399340/ [7399342] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:34 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399342/ [7399343] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:17:22 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399343/ [7399344] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:51:20 info: http://pulpito.ceph.com/vshankar-2023-09-20_10:42:39-fs-wip-vshankar-testing-20230920.072635-testing-default-smithi/7399344/