2023-09-11T09:49:13.023 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-11T09:49:13.024 INFO:teuthology.results:subset: '623/1024' 2023-09-11T09:49:13.024 INFO:teuthology.results:seed: '297' 2023-09-11T14:31:08.793 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-11T14:31:08.819 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-12T02:19:07.213 INFO:teuthology.results:Tests finished! gathering results... 2023-09-12T02:19:26.892 INFO:scrape:Found 202 jobs 2023-09-12T02:19:29.397 WARNING:scrape:Misunderstood line: 2023-09-12T02:19:35.029 WARNING:scrape:Misunderstood line: 2023-09-12T02:20:07.511 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/teuthology.log 2023-09-12T02:20:07.513 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/teuthology.log 2023-09-12T02:20:40.047 INFO:scrape:Found 23 distinct failure reasons 2023-09-12T02:20:40.048 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-09-12T02:20:40.049 INFO:scrape:1 jobs: ['7393538'] 2023-09-12T02:20:40.049 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-09-12T02:20:40.050 INFO:scrape: 2023-09-12T02:20:40.050 INFO:scrape:Failure: Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6ebdfb72-5098-11ee-9ab7-7b867c8bd7da -e sha1=02cf814a2fe196b91c36545e55f7fa303a1a5367 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-09-12T02:20:40.051 INFO:scrape:2 jobs: ['7393451', '7393527'] 2023-09-12T02:20:40.051 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-12T02:20:40.052 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-12T02:20:40.052 INFO:scrape: 2023-09-12T02:20:40.053 INFO:scrape:Failure: Command failed on smithi151 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-09-12T02:20:40.053 INFO:scrape:3 jobs: ['7393452', '7393552', '7393434'] 2023-09-12T02:20:40.054 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-09-12T02:20:40.054 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'tasks/mdtest}', 'validater/valgrind}'] 2023-09-12T02:20:40.055 INFO:scrape: 2023-09-12T02:20:40.055 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-12T02:20:40.056 INFO:scrape:1 jobs: ['7393512'] 2023-09-12T02:20:40.056 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-09-12T02:20:40.057 INFO:scrape: 2023-09-12T02:20:40.057 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-09-12T02:20:40.058 INFO:scrape:1 jobs: ['7393549'] 2023-09-12T02:20:40.058 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-09-12T02:20:40.059 INFO:scrape: 2023-09-12T02:20:40.059 INFO:scrape:Failure: No module named 'tasks' 2023-09-12T02:20:40.059 INFO:scrape:3 jobs: ['7393483', '7393471', '7393484'] 2023-09-12T02:20:40.060 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-12T02:20:40.060 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/iozone}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fail_fs/yes', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/fsstress', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{with-quota}', 'validater/lockdep}', 'wsync/no}'] 2023-09-12T02:20:40.061 INFO:scrape: 2023-09-12T02:20:40.061 INFO:scrape:Crash: Test failure: test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover), test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover) 2023-09-12T02:20:40.061 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9265bb6420] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x55b1289ba973] 3: (Beacon::notify_health(MDSRank const*)+0x2321) [0x55b12866d6b1] 4: (Context::complete(int)+0xd) [0x55b12861dadd] 5: (CommonSafeTimer::timer_thread()+0x18e) [0x7f926601b86e] 6: (CommonSafeTimerThread::entry()+0x11) [0x7f926601cde1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9265baa609] 2023-09-12T02:20:40.062 INFO:scrape:4 jobs: ['7393453', '7393500', '7393443', '7393529'] 2023-09-12T02:20:40.062 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-12T02:20:40.063 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'tasks/failover}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-12T02:20:40.063 INFO:scrape: 2023-09-12T02:20:40.064 INFO:scrape:Dead: b'2023-09-12T01:53:55.330 DEBUG:teuthology.exit:Finished running handlers' 2023-09-12T02:20:40.064 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x560f0fef0d53] 2: (KernelDevice::_aio_thread()+0x1686) [0x560f10b30136] 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x560f10b35b51] 4: /lib64/libpthread.so.0(+0x81ca) [0x7fd3970081ca] 2023-09-12T02:20:40.064 INFO:scrape:1 jobs: ['7393556'] 2023-09-12T02:20:40.066 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-09-12T02:20:40.066 INFO:scrape: 2023-09-12T02:20:40.067 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-09-12T02:20:40.067 INFO:scrape:1 jobs: ['7393390'] 2023-09-12T02:20:40.068 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-09-12T02:20:40.068 INFO:scrape: 2023-09-12T02:20:40.069 INFO:scrape:Failure: "2023-09-11T10:14:37.456922+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-09-12T02:20:40.069 INFO:scrape:5 jobs: ['7393382', '7393460', '7393514', '7393533', '7393469'] 2023-09-12T02:20:40.069 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-12T02:20:40.070 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'session_timeout', 'supported-random-distros$/{centos_latest}', 'tasks/forward-scrub}', 'tasks/mirror}', 'tasks/scrub}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-12T02:20:40.070 INFO:scrape: 2023-09-12T02:20:40.071 INFO:scrape:Dead: b'2023-09-11T23:42:32.704 DEBUG:teuthology.exit:Finished running handlers' 2023-09-12T02:20:40.071 INFO:scrape:4 jobs: ['7393467', '7393394', '7393385', '7393567'] 2023-09-12T02:20:40.071 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-09-12T02:20:40.072 INFO:scrape:suites union: ['1-ceph', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fail_fs/no', 'fail_fs/yes', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/failover}', 'tasks/xfstests-dev}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2023-09-12T02:20:40.072 INFO:scrape: 2023-09-12T02:20:40.073 INFO:scrape:Crash: Test failure: test_extra_data_pool_rebuild_few_files (tasks.cephfs.test_data_scan.TestDataScan) 2023-09-12T02:20:40.073 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fb9c6c12cf0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x5596d68824c3] 3: (Beacon::notify_health(MDSRank const*)+0x21cf) [0x5596d6527daf] 4: (MDSRank::damaged()+0x53) [0x5596d64de2e3] 5: (MDSTable::load_2(int, ceph::buffer::v15_2_0::list&, Context*)+0x351) [0x5596d6814701] 6: (MDSContext::complete(int)+0x5f) [0x5596d68717cf] 7: (MDSIOContextBase::complete(int)+0x574) [0x5596d6871fb4] 8: (Finisher::finisher_thread_entry()+0x19d) [0x7fb9c8953e4d] 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb9c6c081ca] 2023-09-12T02:20:40.073 INFO:scrape:2 jobs: ['7393450', '7393468'] 2023-09-12T02:20:40.074 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health'] 2023-09-12T02:20:40.074 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}', 'tasks/journal-repair}'] 2023-09-12T02:20:40.075 INFO:scrape: 2023-09-12T02:20:40.075 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount', 'mds_server.req_mksnap_latency.avgcount', 'mds.root_rsnaps'} 2023-09-12T02:20:40.076 INFO:scrape:5 jobs: ['7393531', '7393404', '7393442', '7393423', '7393572'] 2023-09-12T02:20:40.076 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2023-09-12T02:20:40.076 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-12T02:20:40.077 INFO:scrape: 2023-09-12T02:20:40.077 INFO:scrape:Crash: Command failed (workunit test fs/misc/trivial_sync.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh' 2023-09-12T02:20:40.078 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fcea419ace0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x56202c9154c3] 3: (Beacon::notify_health(MDSRank const*)+0x21cf) [0x56202c5badaf] 4: (Context::complete(int)+0xd) [0x56202c5676cd] 5: (CommonSafeTimer::timer_thread()+0x181) [0x7fcea54efc91] 6: (CommonSafeTimerThread::entry()+0x11) [0x7fcea54f0fa1] 7: /lib64/libpthread.so.0(+0x81cf) [0x7fcea41901cf] 2023-09-12T02:20:40.078 INFO:scrape:2 jobs: ['7393493', '7393523'] 2023-09-12T02:20:40.079 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/stock/{k-stock', 'rhel_8}'] 2023-09-12T02:20:40.079 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/failover}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-12T02:20:40.080 INFO:scrape: 2023-09-12T02:20:40.080 INFO:scrape:Failure: Command failed on smithi130 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6013-g02cf814a-1focal ceph-mds=18.0.0-6013-g02cf814a-1focal ceph-common=18.0.0-6013-g02cf814a-1focal ceph-fuse=18.0.0-6013-g02cf814a-1focal ceph-test=18.0.0-6013-g02cf814a-1focal radosgw=18.0.0-6013-g02cf814a-1focal python-ceph=18.0.0-6013-g02cf814a-1focal libcephfs1=18.0.0-6013-g02cf814a-1focal libcephfs-java=18.0.0-6013-g02cf814a-1focal libcephfs-jni=18.0.0-6013-g02cf814a-1focal librados2=18.0.0-6013-g02cf814a-1focal librbd1=18.0.0-6013-g02cf814a-1focal rbd-fuse=18.0.0-6013-g02cf814a-1focal python3-cephfs=18.0.0-6013-g02cf814a-1focal cephfs-shell=18.0.0-6013-g02cf814a-1focal cephfs-top=18.0.0-6013-g02cf814a-1focal cephfs-mirror=18.0.0-6013-g02cf814a-1focal' 2023-09-12T02:20:40.080 INFO:scrape:1 jobs: ['7393474'] 2023-09-12T02:20:40.081 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-09-12T02:20:40.081 INFO:scrape: 2023-09-12T02:20:40.082 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/iozone.sh 2023-09-12T02:20:40.082 INFO:scrape:1 jobs: ['7393555'] 2023-09-12T02:20:40.082 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-12T02:20:40.083 INFO:scrape: 2023-09-12T02:20:40.083 INFO:scrape:Failure: Command failed on smithi184 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ecd8e44-50b1-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json' 2023-09-12T02:20:40.084 INFO:scrape:1 jobs: ['7393546'] 2023-09-12T02:20:40.084 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-09-12T02:20:40.084 INFO:scrape: 2023-09-12T02:20:40.085 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-09-12T02:20:40.085 INFO:scrape:1 jobs: ['7393503'] 2023-09-12T02:20:40.086 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-09-12T02:20:40.086 INFO:scrape: 2023-09-12T02:20:40.087 INFO:scrape:Crash: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2023-09-12T02:20:40.088 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f95f6925afb] 2: (CDentry::check_corruption(bool)+0x6fc) [0x5630e797cc3c] 3: ceph-mds(+0x47d7a2) [0x5630e79967a2] 4: (CDir::_omap_commit(int)+0x30a) [0x5630e7996d5a] 5: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x12d) [0x5630e7997f7d] 6: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x257e) [0x5630e773fabe] 7: (MDLog::try_expire(LogSegment*, int)+0x68) [0x5630e7a6f828] 8: (MDLog::trim_all()+0x6cd) [0x5630e7a70fcd] 9: (C_Flush_Journal::trim_mdlog()+0x14b) [0x5630e770f13b] 10: (Context::complete(int)+0xd) [0x5630e76cbadd] 11: (MDSContext::complete(int)+0x5f) [0x5630e7a581af] 12: (C_IO_Wrapper::finish(int)+0x12) [0x5630e7a59062] 13: (MDSContext::complete(int)+0x5f) [0x5630e7a581af] 14: (MDSIOContextBase::complete(int)+0x55c) [0x5630e7a5896c] 15: (Finisher::finisher_thread_entry()+0x195) [0x7f95f69cdde5] 16: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f95f73b0609] 2023-09-12T02:20:40.088 INFO:scrape:1 jobs: ['7393447'] 2023-09-12T02:20:40.088 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-09-12T02:20:40.089 INFO:scrape: 2023-09-12T02:20:40.089 INFO:scrape:Dead: None 2023-09-12T02:20:40.090 INFO:scrape:1 jobs: ['7393458'] 2023-09-12T02:20:40.090 INFO:scrape: 2023-09-12T02:20:40.090 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2023-09-12T02:20:40.091 INFO:scrape:2 jobs: ['7393396', '7393564'] 2023-09-12T02:20:40.091 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-09-12T02:20:40.092 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-09-12T02:20:40.092 INFO:scrape: 2023-09-12T02:20:40.092 INFO:scrape:Failure: Test failure: test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins) 2023-09-12T02:20:40.093 INFO:scrape:1 jobs: ['7393457'] 2023-09-12T02:20:40.093 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] 2023-09-12T02:20:40.094 INFO:scrape: 2023-09-12T02:20:40.094 INFO:scrape:Crash: Command failed on smithi094 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0' 2023-09-12T02:20:40.094 INFO:scrape:ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f708e454db0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x55ed75cfd5f3] 3: (Beacon::notify_health(MDSRank const*)+0x245a) [0x55ed75a87a1a] 4: ceph-mds(+0x123d9d) [0x55ed75a1dd9d] 5: (CommonSafeTimer::timer_thread()+0x15e) [0x7f708ee54dce] 6: /usr/lib64/ceph/libceph-common.so.2(+0x255691) [0x7f708ee55691] 7: /lib64/libc.so.6(+0x9f802) [0x7f708e49f802] 8: /lib64/libc.so.6(+0x3f450) [0x7f708e43f450] 2023-09-12T02:20:40.095 INFO:scrape:1 jobs: ['7393417'] 2023-09-12T02:20:40.095 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-12T02:20:40.096 INFO:scrape: 2023-09-12T02:20:40.106 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/ failed: 39 dead: 6 running: 0 waiting: 0 queued: 0 passed: 157 Fail ================================================================= [7393382] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:18:06 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393382/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393382/ "2023-09-11T10:14:37.456922+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7393390] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:30:46 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393390/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8640246a2c7458a840945f32a3851a0 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7393396] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:17:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393396/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0b6b67c20ba4b3c816a22019320c5c0 Command failed (workunit test fs/quota/quota.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' [7393404] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:08:48 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393404/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393404/ The following counters failed to be set on mds daemons: {'mds.root_rsnaps', 'mds_server.req_rmsnap_latency.avgcount', 'mds_server.req_mksnap_latency.avgcount'} [7393417] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:24:53 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393417/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd3e0add9b8749d986a681183e1668b9 Command failed on smithi094 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0' [7393423] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:39:32 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393423/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393423/ The following counters failed to be set on mds daemons: {'mds_server.req_mksnap_latency.avgcount', 'mds.root_rsnaps', 'mds_server.req_rmsnap_latency.avgcount'} [7393434] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:56:28 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393434/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393434/ Command failed on smithi026 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7393442] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:39:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393442/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393442/ The following counters failed to be set on mds daemons: {'mds_server.req_mksnap_latency.avgcount', 'mds.root_rsnaps', 'mds_server.req_rmsnap_latency.avgcount'} [7393443] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:26:53 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393443/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=60dadb2276c54025ad8197d2b18a9183 Command failed (workunit test suites/pjd.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7393447] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:21:49 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393447/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393447/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f022eb779a2d4a088d82c8aadedbaa49 Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) [7393450] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:18:41 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393450/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393450/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a937e503bc854042a1364f39d70860c2 Test failure: test_extra_data_pool_rebuild_few_files (tasks.cephfs.test_data_scan.TestDataScan) [7393451] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:28:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393451/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3edcb8c7ab7241cf868814ca71ea557a Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6ebdfb72-5098-11ee-9ab7-7b867c8bd7da -e sha1=02cf814a2fe196b91c36545e55f7fa303a1a5367 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7393452] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393452/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393452/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13caeed3d6a84f1cab651a7392c2a6cd Command failed on smithi151 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7393453] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:38:34 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393453/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edd16b11a64f4bd9a316e0c847c03e2e Test failure: test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover), test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover) [7393457] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:33:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393457/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=549eb3878eda499b881023f4a7b5f03d Test failure: test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins) [7393460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:18:06 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393460/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393460/ "2023-09-11T11:55:59.748252+0000 mds.c (mds.0) 1 : cluster [WRN] _replay status 100% complete Estimated time remaining: 0 seconds" in cluster log [7393468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:20:29 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393468/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393468/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82a8b73e5afc49e0ad78de80a1350f72 Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) [7393469] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:20:44 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393469/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393469/ "2023-09-11T12:03:59.628157+0000 mds.b (mds.0) 1 : cluster [WRN] _replay status 73.8059% complete Estimated time remaining: 0 seconds" in cluster log [7393471] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:19:38 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393471/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f94c298f99545959abb76e51f65fa98 No module named 'tasks.kclient' [7393474] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393474/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9029ca537c7a4f65a0f392862bcb9ae7 Command failed on smithi130 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6013-g02cf814a-1focal ceph-mds=18.0.0-6013-g02cf814a-1focal ceph-common=18.0.0-6013-g02cf814a-1focal ceph- fuse=18.0.0-6013-g02cf814a-1focal ceph-test=18.0.0-6013-g02cf814a-1focal radosgw=18.0.0-6013-g02cf814a-1focal python- ceph=18.0.0-6013-g02cf814a-1focal libcephfs1=18.0.0-6013-g02cf814a-1focal libcephfs-java=18.0.0-6013-g02cf814a-1focal libcephfs- jni=18.0.0-6013-g02cf814a-1focal librados2=18.0.0-6013-g02cf814a-1focal librbd1=18.0.0-6013-g02cf814a-1focal rbd-fuse=18.0.0-6013-g02cf814a-1focal python3-cephfs=18.0.0-6013-g02cf814a-1focal cephfs- shell=18.0.0-6013-g02cf814a-1focal cephfs-top=18.0.0-6013-g02cf814a-1focal cephfs-mirror=18.0.0-6013-g02cf814a-1focal' [7393483] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393483/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=feaa3453db1141e5b44b24c996251c4a No module named 'tasks' [7393484] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:15:48 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393484/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b8a57f7944d4c8784c7e5b80e4d92fe No module named 'tasks' [7393493] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:23:28 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393493/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3e044d5336d46ff8ea875a06a938456 Command failed (workunit test fs/misc/trivial_sync.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh' [7393500] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:43:26 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393500/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f1660f2b4d74d5c8f0febb6ee088dbb Command failed (workunit test suites/ffsb.sh) on smithi061 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7393503] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:16:06 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393503/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2339a1219b64889ab0c6ff1d479bb0b Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7393512] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:31:03 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393512/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e8dc7971702401496e1afd39825f613 saw valgrind issues [7393514] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:29:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393514/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393514/ "2023-09-11T13:28:07.530244+0000 mds.c (mds.0) 1 : cluster [WRN] _replay status 100% complete Estimated time remaining: 0 seconds" in cluster log [7393523] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:37:45 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393523/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6edf279e05b641eab441ed32eda1eb48 Test failure: test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover), test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover) [7393527] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:28:38 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393527/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=643163902fdb4412bbc87fd1edeab01e Command failed on smithi151 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03e170ba-50af-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json' [7393529] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393529/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19569861a7854d178ef75a99bdbb3238 Command failed (workunit test suites/ffsb.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7393531] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:50:56 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393531/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393531/ The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount', 'mds_server.req_mksnap_latency.avgcount', 'mds.root_rsnaps'} [7393533] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:25:46 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393533/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393533/ "2023-09-11T13:38:06.511627+0000 mds.d (mds.0) 1 : cluster [WRN] _replay status 100% complete Estimated time remaining: 4 seconds" in cluster log [7393538] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:19:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393538/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=447e2c73d52c49d0bacb43d73783d957 Command failed (workunit test fs/test_python.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7393546] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:30:41 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393546/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393546/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c50ca813e1184a1d95c8c0b586f6956d Command failed on smithi184 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ecd8e44-50b1-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json' [7393549] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:46:21 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393549/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393549/ error during scrub thrashing: rank damage found: {'backtrace'} [7393552] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393552/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393552/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea349f91f3fc4af4b3971804c8430f27 Command failed on smithi161 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7393555] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 03:15:58 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393555/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393555/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64124d2e474a47b793d9562c6c4b7836 Command failed (workunit test suites/iozone.sh) on smithi027 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [7393564] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:21:12 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393564/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393564/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abfdcf68655d4062ac8f08062d0ccb4b Command failed (workunit test fs/quota/quota.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7393572] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:46:11 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393572/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393572/ The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount', 'mds.root_rsnaps', 'mds_server.req_mksnap_latency.avgcount'} Dead ================================================================= [7393385] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393385/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393385/ hit max job timeout [7393394] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393394/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393394/ hit max job timeout [7393458] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7393467] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393467/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393467/ hit max job timeout [7393556] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393556/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393556/ hit max job timeout [7393567] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393567/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393567/ hit max job timeout Pass ================================================================= [7393375] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:06 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393375/ [7393376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:21:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393376/ [7393377] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:11 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393377/ [7393378] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:16:05 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393378/ [7393379] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:50:45 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393379/ [7393380] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:23:46 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393380/ [7393381] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:17:10 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393381/ [7393383] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit} time: 00:34:48 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393383/ [7393384] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:02:13 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393384/ [7393386] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:23:17 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393386/ [7393387] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:49:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393387/ [7393388] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:13:47 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393388/ [7393389] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:17:13 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393389/ [7393391] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:11 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393391/ [7393392] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:26:51 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393392/ [7393393] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:42:10 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393393/ [7393395] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:21 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393395/ [7393397] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:07 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393397/ [7393398] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:22:13 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393398/ [7393399] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:17:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393399/ [7393400] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:16:29 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393400/ [7393401] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:43:26 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393401/ [7393402] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} time: 02:27:54 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393402/ [7393403] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:24:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393403/ [7393405] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:29 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393405/ [7393406] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:39:06 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393406/ [7393407] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:48:57 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393407/ [7393408] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393408/ [7393409] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393409/ [7393410] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:16:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393410/ [7393411] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393411/ [7393412] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:53 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393412/ [7393413] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:43:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393413/ [7393414] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393414/ [7393415] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:03 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393415/ [7393416] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:18:34 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393416/ [7393418] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:16:22 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393418/ [7393419] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:39 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393419/ [7393420] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:16:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393420/ [7393421] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:21 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393421/ [7393422] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:47:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393422/ [7393424] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:43:35 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393424/ [7393425] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:53 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393425/ [7393426] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:24:03 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393426/ [7393427] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393427/ [7393428] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:13:50 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393428/ [7393429] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:41:19 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393429/ [7393430] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393430/ [7393431] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:23:58 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393431/ [7393432] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:35:43 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393432/ [7393433] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:17:56 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393433/ [7393435] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:54 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393435/ [7393436] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:15:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393436/ [7393437] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:37 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393437/ [7393438] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:16:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393438/ [7393439] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393439/ [7393440] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:21:25 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393440/ [7393441] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:50:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393441/ [7393444] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:26 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393444/ [7393445] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:34 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393445/ [7393446] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393446/ [7393448] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:32:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393448/ [7393449] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:51:10 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393449/ [7393454] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:41:26 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393454/ [7393455] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:51:02 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393455/ [7393456] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:25:54 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393456/ [7393459] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:59 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393459/ [7393461] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393461/ [7393462] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:59 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393462/ [7393463] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393463/ [7393464] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:23:01 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393464/ [7393465] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:16 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393465/ [7393466] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:47:34 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393466/ [7393470] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:15:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393470/ [7393472] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:51 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393472/ [7393473] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:46:49 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393473/ [7393475] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:29:51 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393475/ [7393476] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:57:00 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393476/ [7393477] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:13:19 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393477/ [7393478] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:13:46 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393478/ [7393479] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:32:53 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393479/ [7393480] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:51:44 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393480/ [7393481] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:14:15 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393481/ [7393482] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393482/ [7393485] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:37:58 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393485/ [7393486] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:56 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393486/ [7393487] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:23:44 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393487/ [7393488] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:31:24 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393488/ [7393489] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:03:07 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393489/ [7393490] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:37:11 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393490/ [7393491] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:02 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393491/ [7393492] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 02:05:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393492/ [7393494] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:25:41 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393494/ [7393495] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:42 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393495/ [7393496] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:39 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393496/ [7393497] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393497/ [7393498] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} time: 00:28:35 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393498/ [7393499] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:30:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393499/ [7393501] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:16:18 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393501/ [7393502] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:20:12 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393502/ [7393504] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:50:28 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393504/ [7393505] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:27 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393505/ [7393506] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:39:25 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393506/ [7393507] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:34:38 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393507/ [7393508] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:01 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393508/ [7393509] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:40 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393509/ [7393510] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:39 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393510/ [7393511] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:34 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393511/ [7393513] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:45:30 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393513/ [7393515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:55:13 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393515/ [7393516] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} time: 02:01:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393516/ [7393517] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:56 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393517/ [7393518] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:48:44 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393518/ [7393519] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:19:20 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393519/ [7393520] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:50 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393520/ [7393525] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:05 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393525/ [7393526] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:49:23 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393526/ [7393528] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:50:05 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393528/ [7393530] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:40:14 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393530/ [7393532] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:14:26 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393532/ [7393534] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:41:51 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393534/ [7393535] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:39 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393535/ [7393536] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:42:31 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393536/ [7393537] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:29:51 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393537/ [7393539] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:14 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393539/ [7393540] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393540/ [7393541] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:32:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393541/ [7393542] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 00:56:50 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393542/ [7393543] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:49:28 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393543/ [7393544] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:35:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393544/ [7393545] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:49 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393545/ [7393547] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:19:08 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393547/ [7393548] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:23:56 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393548/ [7393550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:15:41 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393550/ [7393551] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:32:41 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393551/ [7393553] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:30:19 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393553/ [7393554] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:56:25 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393554/ [7393557] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:02 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393557/ [7393558] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:22:20 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393558/ [7393559] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:45:59 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393559/ [7393560] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393560/ [7393561] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393561/ [7393562] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:39:10 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393562/ [7393563] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 01:05:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393563/ [7393565] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:21:36 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393565/ [7393566] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:50 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393566/ [7393568] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:22:04 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393568/ [7393569] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:21:31 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393569/ [7393570] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:20:54 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393570/ [7393571] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:54 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393571/ [7393573] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:19:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393573/ [7393574] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:40:22 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393574/ [7393575] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:42:55 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393575/ [7393576] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:23:21 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393576/ [7393577] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:51:44 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393577/ [7393578] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:27:52 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393578/ [7393579] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:33 info: http://pulpito.ceph.com/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393579/