2022-10-19T06:52:32.059 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-19T06:52:32.063 INFO:teuthology.results:subset: '150/256' 2022-10-19T06:52:32.063 INFO:teuthology.results:seed: '432' 2022-10-20T06:00:49.563 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-20T06:00:49.589 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-20T08:04:01.015 INFO:teuthology.results:Tests finished! gathering results... 2022-10-20T08:04:27.560 INFO:scrape:Found 241 jobs 2022-10-20T08:05:12.785 INFO:scrape:Found 18 distinct failure reasons 2022-10-20T08:05:12.787 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2022-10-20T08:05:12.788 INFO:scrape:6 jobs: ['7073227', '7073139', '7073078', '7073020', '7073036', '7073232'] 2022-10-20T08:05:12.788 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-10-20T08:05:12.789 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_latest', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-10-20T08:05:12.789 INFO:scrape: 2022-10-20T08:05:12.790 INFO:scrape:Failure: No module named 'tasks.ceph' 2022-10-20T08:05:12.790 INFO:scrape:4 jobs: ['7073149', '7073151', '7073161', '7073154'] 2022-10-20T08:05:12.791 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2022-10-20T08:05:12.791 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client-upgrade', '2-client-sanity}}', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/fsync-tester}}', 'begin/{0-install', 'centos_latest', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_latest', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/truncate_delay}', 'tasks/workunit/dir-max-entries}', 'tasks/{0-nautilus', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2022-10-20T08:05:12.792 INFO:scrape: 2022-10-20T08:05:12.792 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2022-10-20T08:05:12.793 INFO:scrape:2 jobs: ['7073014', '7073066'] 2022-10-20T08:05:12.793 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2022-10-20T08:05:12.794 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'centos_8', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu/{latest', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}'] 2022-10-20T08:05:12.794 INFO:scrape: 2022-10-20T08:05:12.794 INFO:scrape:Failure: Command failed on smithi111 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 637205ea-4f7e-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" 2022-10-20T08:05:12.795 INFO:scrape:6 jobs: ['7073000', '7073082', '7073116', '7073067', '7073039', '7073179'] 2022-10-20T08:05:12.796 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2022-10-20T08:05:12.796 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2022-10-20T08:05:12.797 INFO:scrape: 2022-10-20T08:05:12.797 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-10-20T08:05:12.798 INFO:scrape:1 jobs: ['7073132'] 2022-10-20T08:05:12.798 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-10-20T08:05:12.798 INFO:scrape: 2022-10-20T08:05:12.799 INFO:scrape:Failure: Test failure: test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) 2022-10-20T08:05:12.799 INFO:scrape:1 jobs: ['7073103'] 2022-10-20T08:05:12.800 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2022-10-20T08:05:12.800 INFO:scrape: 2022-10-20T08:05:12.801 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2022-10-20T08:05:12.801 INFO:scrape:2 jobs: ['7072995', '7073192'] 2022-10-20T08:05:12.802 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2022-10-20T08:05:12.802 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-10-20T08:05:12.803 INFO:scrape: 2022-10-20T08:05:12.803 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi033 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-10-20T08:05:12.804 INFO:scrape:ceph version 18.0.0-530-g3b3779af (3b3779af3af21841d0a5958b774b3b9b62d61caa) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9ebfe12ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f9ec12a6e1f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a6f8b) [0x7f9ec12a6f8b] 6: (MetaSession::~MetaSession()+0x1eb) [0x55c06f2ba8cb] 7: (std::_Sp_counted_ptr::_M_dispose()+0x16) [0x55c06f2ba946] 8: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x3a) [0x55c06f29cd1a] 9: (Client::handle_client_session(boost::intrusive_ptr const&)+0x100) [0x55c06f283270] 10: (Client::ms_dispatch2(boost::intrusive_ptr const&)+0x555) [0x55c06f286755] 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9ec1513968] 12: (DispatchQueue::entry()+0x50f) [0x7f9ec1510b0f] 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9ec15d5691] 14: /lib64/libpthread.so.0(+0x81cf) [0x7f9ebfe081cf] 15: clone() 2022-10-20T08:05:12.804 INFO:scrape:1 jobs: ['7073187'] 2022-10-20T08:05:12.805 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-10-20T08:05:12.805 INFO:scrape: 2022-10-20T08:05:12.806 INFO:scrape:Failure: "1666174093.0143821 mon.a (mon.0) 514 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2022-10-20T08:05:12.806 INFO:scrape:2 jobs: ['7073113', '7073072'] 2022-10-20T08:05:12.807 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health'] 2022-10-20T08:05:12.807 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_latest', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/metrics}', 'tasks/snap-schedule}'] 2022-10-20T08:05:12.808 INFO:scrape: 2022-10-20T08:05:12.808 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) 2022-10-20T08:05:12.808 INFO:scrape:1 jobs: ['7073054'] 2022-10-20T08:05:12.809 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-flush}'] 2022-10-20T08:05:12.809 INFO:scrape: 2022-10-20T08:05:12.810 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-10-20T08:05:12.810 INFO:scrape:4 jobs: ['7073073', '7073027', '7073117', '7073221'] 2022-10-20T08:05:12.811 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'n/3', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay'] 2022-10-20T08:05:12.811 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_latest', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-10-20T08:05:12.812 INFO:scrape: 2022-10-20T08:05:12.812 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi035.front.sepia.ceph.com: ['type=AVC msg=audit(1666185655.892:192): avc: denied { node_bind } for pid=1414 comm="ping" saddr=172.21.15.35 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-10-20T08:05:12.813 INFO:scrape:1 jobs: ['7073209'] 2022-10-20T08:05:12.813 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2022-10-20T08:05:12.813 INFO:scrape: 2022-10-20T08:05:12.814 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/fsstress.sh 2022-10-20T08:05:12.814 INFO:scrape:2 jobs: ['7073195', '7073101'] 2022-10-20T08:05:12.815 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2022-10-20T08:05:12.815 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2022-10-20T08:05:12.816 INFO:scrape: 2022-10-20T08:05:12.816 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2022-10-20T08:05:12.817 INFO:scrape:1 jobs: ['7073145'] 2022-10-20T08:05:12.817 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}'] 2022-10-20T08:05:12.818 INFO:scrape: 2022-10-20T08:05:12.818 INFO:scrape:Failure: Command failed on smithi138 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3b3779af3af21841d0a5958b774b3b9b62d61caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18e1dd10-4f98-11ed-8437-001a4aab830c -- ceph orch host add smithi150' 2022-10-20T08:05:12.818 INFO:scrape:1 jobs: ['7073121'] 2022-10-20T08:05:12.819 INFO:scrape:suites: ['1-mount', '2-workload/dbench', 'distro/single-container-host}', 'fs/cephadm/multivolume/{0-start'] 2022-10-20T08:05:12.819 INFO:scrape: 2022-10-20T08:05:12.820 INFO:scrape:Failure: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2022-10-20T08:05:12.820 INFO:scrape:1 jobs: ['7073021'] 2022-10-20T08:05:12.821 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] 2022-10-20T08:05:12.821 INFO:scrape: 2022-10-20T08:05:12.821 INFO:scrape:Failure: 2022-10-20T08:05:12.822 INFO:scrape:1 jobs: ['7073176'] 2022-10-20T08:05:12.822 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}'] 2022-10-20T08:05:12.823 INFO:scrape: 2022-10-20T08:05:12.823 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2022-10-20T08:05:12.824 INFO:scrape:1 jobs: ['7073040'] 2022-10-20T08:05:12.824 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-10-20T08:05:12.825 INFO:scrape: 2022-10-20T08:05:12.850 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/ failed: 38 dead: 0 running: 0 waiting: 0 queued: 0 passed: 203 Fail ================================================================= [7072995] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:42:55 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072995/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072995/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08109f597eb64ab581c102b5f0ce37bd Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7073000] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073000/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073000/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6f54eb381a9466f8bc4b0f49357e122 Command failed on smithi111 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 637205ea-4f7e-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7073014] fs/verify/{begin/{0-install 1-ceph 2-logrotate} centos_8 clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 06:17:14 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073014/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073014/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3cfd738ef4445f5af0696488fbcce1f Command failed (workunit test suites/fsstress.sh) on smithi120 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7073020] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 01:15:14 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073020/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073020/ error during scrub thrashing: rank damage found: {'backtrace'} [7073021] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:22:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073021/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=640ece63ed8e428baff5925331eefc79 Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) [7073027] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:23:32 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073027/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073027/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d8cf68ec1f64911b78eef9ccb24f6d3 Command failed (workunit test suites/pjd.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7073036] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:44:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073036/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073036/ error during scrub thrashing: rank damage found: {'backtrace'} [7073039] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:16:52 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073039/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073039/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f5d2bcd444147ff95c8808eab5c8cb2 Command failed on smithi083 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cfdf4010-4f85-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7073040] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 03:16:20 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073040/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073040/ Exiting scrub checking -- not all pgs scrubbed. [7073054] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073054/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=906b2d737874406ba2033bb8eb85cc16 Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) [7073066] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:14:42 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073066/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073066/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1def846ca695489691e683a9ac10f803 Command failed (workunit test suites/fsstress.sh) on smithi143 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7073067] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:19:51 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073067/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f45453d0df7a4120b2916b7862e723a1 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0bd81096-4f8c-11ed-8437-001a4aab830c -- ceph mon dump -f json' [7073072] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:40:42 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073072/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073072/ "1666171116.5211608 mon.a (mon.0) 877 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7073073] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:27:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073073/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073073/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d5047478bb4450a9598bd7ca444d9d3 Command failed (workunit test suites/pjd.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7073078] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:42:22 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073078/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073078/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7073082] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073082/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073082/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a4f0d19f7784f3eb7cdbb28230d9b11 Command failed on smithi149 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c581cba2-4f8e-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7073101] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:42:17 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073101/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e295affdbb7470191e6a14a4ace0a9c Command failed (workunit test suites/fsstress.sh) on smithi055 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7073103] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:20:42 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073103/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=186b1c862ab147b6820868b51d21371a Test failure: test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) [7073113] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:45:33 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073113/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073113/ "1666174093.0143821 mon.a (mon.0) 514 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7073116] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:31 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073116/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073116/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20899af3bbd34d75a5594f80f7763c45 Command failed on smithi027 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c763152-4f97-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7073117] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:10 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073117/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073117/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48d654c753eb4a199545b69bd296215f Command failed (workunit test suites/pjd.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7073121] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073121/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3a07fe7369f4b619d3eaec0ac271e04 Command failed on smithi138 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:3b3779af3af21841d0a5958b774b3b9b62d61caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18e1dd10-4f98-11ed-8437-001a4aab830c -- ceph orch host add smithi150' [7073132] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:32:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073132/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073132/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9aaff5a10eee4a5aaa339cf6a9e7d305 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7073139] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 04:06:49 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073139/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073139/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7073145] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:17:22 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073145/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073145/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2b55e948209420ca7b8fcf2da9a71ea Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [7073149] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:07:23 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073149/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a912019cecd4be0b80363c936bada52 No module named 'tasks.ceph' [7073151] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:07:19 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073151/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=022d382ec77c4d1d9a10bae04dfbfb12 No module named 'tasks.ceph' [7073154] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:05:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073154/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f39e6f9affdc401ba880f7528183b6d2 No module named 'tasks.ceph' [7073161] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:07:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073161/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073161/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7123a7c6edf342f39b26c51589fa8d7f No module named 'tasks' [7073176] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:55 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba43cc5eb6944f1d88c3e18cc6c6b1b8 [7073179] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:21 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a35ecb1a23f3475f98ead2942f42610a Command failed on smithi043 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8fb39ad8-4fab-11ed-8437-001a4aab830c -e sha1=3b3779af3af21841d0a5958b774b3b9b62d61caa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7073187] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:17:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073187/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c386ba9778b45c1a436fd42d72e110b Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi033 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7073192] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:35:45 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073192/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=474e62dc547c46e7b9791fa10b96035d Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7073195] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:57:02 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fd470bb811143b3afc5aeb43e30443e Command failed (workunit test suites/fsstress.sh) on smithi061 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7073209] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:59:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073209/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073209/ SELinux denials found on ubuntu@smithi035.front.sepia.ceph.com: ['type=AVC msg=audit(1666185655.892:192): avc: denied { node_bind } for pid=1414 comm="ping" saddr=172.21.15.35 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7073221] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073221/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073221/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dad693aa50814855872e2af65678b8b0 Command failed (workunit test suites/pjd.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b3779af3af21841d0a5958b774b3b9b62d61caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7073227] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:43:44 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073227/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073227/ error during scrub thrashing: rank damage found: {'backtrace'} [7073232] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:57:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073232/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073232/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Pass ================================================================= [7072994] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072994/ [7072996] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:08:24 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072996/ [7072997] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072997/ [7072998] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 01:14:03 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072998/ [7072999] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:36:37 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7072999/ [7073001] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:20:57 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073001/ [7073002] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:21 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073002/ [7073003] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:20:25 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073003/ [7073004] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:21:55 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073004/ [7073005] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:03:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073005/ [7073006] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:39:38 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073006/ [7073007] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:23:17 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073007/ [7073008] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:33:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073008/ [7073009] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:07:16 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073009/ [7073010] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:24:14 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073010/ [7073011] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:52 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073011/ [7073012] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:33:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073012/ [7073013] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:12:07 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073013/ [7073015] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:13 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073015/ [7073016] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:21:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073016/ [7073017] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:28:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073017/ [7073018] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:01 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073018/ [7073019] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073019/ [7073022] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:10:43 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073022/ [7073023] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:32:58 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073023/ [7073024] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:24:34 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073024/ [7073025] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:24:29 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073025/ [7073026] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073026/ [7073028] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:39:37 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073028/ [7073029] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:32:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073029/ [7073030] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:11:25 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073030/ [7073031] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 02:04:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073031/ [7073032] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:41:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073032/ [7073033] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:32 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073033/ [7073034] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:26:31 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073034/ [7073035] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:35 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073035/ [7073037] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:24:38 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073037/ [7073038] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 01:51:42 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073038/ [7073041] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:02 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073041/ [7073042] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:26:42 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073042/ [7073043] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073043/ [7073044] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:51:25 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073044/ [7073045] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073045/ [7073046] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:16:29 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073046/ [7073047] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 02:00:19 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073047/ [7073048] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:34:53 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073048/ [7073049] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:31 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073049/ [7073050] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:15:23 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073050/ [7073051] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:36:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073051/ [7073052] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:31:01 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073052/ [7073053] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:29:38 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073053/ [7073055] fs/verify/{begin/{0-install 1-ceph 2-logrotate} centos_8 clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:52:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073055/ [7073056] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:22 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073056/ [7073057] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:34:23 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073057/ [7073058] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:10:21 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073058/ [7073059] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:23:26 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073059/ [7073060] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:25:02 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073060/ [7073061] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:31:03 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073061/ [7073062] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:38 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073062/ [7073063] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:11:39 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073063/ [7073064] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:19 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073064/ [7073065] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:08:02 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073065/ [7073068] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:50:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073068/ [7073069] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:31:43 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073069/ [7073070] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:44:40 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073070/ [7073071] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:30:28 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073071/ [7073074] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:21:27 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073074/ [7073075] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:52 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073075/ [7073076] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:29:40 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073076/ [7073077] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:17 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073077/ [7073079] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:12:17 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073079/ [7073080] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:26 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073080/ [7073081] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 01:47:27 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073081/ [7073083] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:01 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073083/ [7073084] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073084/ [7073085] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:10:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073085/ [7073086] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:22 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073086/ [7073087] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 00:43:52 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073087/ [7073088] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:19:59 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073088/ [7073089] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 01:17:40 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073089/ [7073090] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:11:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073090/ [7073091] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 01:08:55 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073091/ [7073092] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:28 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073092/ [7073093] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073093/ [7073094] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:27 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073094/ [7073095] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:13 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073095/ [7073096] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 01:15:08 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073096/ [7073097] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:24:53 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073097/ [7073098] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:08:35 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073098/ [7073099] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:15:28 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073099/ [7073100] fs/verify/{begin/{0-install 1-ceph 2-logrotate} centos_8 clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:18:01 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073100/ [7073102] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} time: 00:34:31 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073102/ [7073104] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:41:21 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073104/ [7073105] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 02:52:58 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073105/ [7073106] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:29:26 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073106/ [7073107] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:18:08 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073107/ [7073108] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073108/ [7073109] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:32:59 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073109/ [7073110] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:28:31 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073110/ [7073111] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:14:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073111/ [7073112] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 01:17:32 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073112/ [7073114] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:18:58 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073114/ [7073115] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:28:19 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073115/ [7073118] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:12:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073118/ [7073119] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073119/ [7073120] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:11:00 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073120/ [7073122] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:27:22 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073122/ [7073123] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:19:25 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073123/ [7073124] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror} time: 02:12:41 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073124/ [7073125] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:35:35 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073125/ [7073126] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:15 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073126/ [7073127] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:38:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073127/ [7073128] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:44 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073128/ [7073129] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:49:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073129/ [7073130] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:03:33 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073130/ [7073131] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:11:02 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073131/ [7073133] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:13:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073133/ [7073134] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:33:03 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073134/ [7073135] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:33:16 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073135/ [7073136] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:32:03 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073136/ [7073137] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} time: 00:50:51 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073137/ [7073138] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:44 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073138/ [7073140] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:26:49 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073140/ [7073141] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:28:15 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073141/ [7073142] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:27:33 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073142/ [7073143] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073143/ [7073144] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:10:27 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073144/ [7073146] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:48:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073146/ [7073147] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:39:35 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073147/ [7073148] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 01:18:29 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073148/ [7073150] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:22:25 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073150/ [7073152] fs/verify/{begin/{0-install 1-ceph 2-logrotate} centos_8 clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:54:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073152/ [7073153] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:35:58 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073153/ [7073155] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:06 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073155/ [7073156] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:29:56 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073156/ [7073157] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:11 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073157/ [7073158] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:34 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073158/ [7073159] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:19:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073159/ [7073160] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:30 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073160/ [7073162] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:44:10 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073162/ [7073163] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073163/ [7073164] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:11:56 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073164/ [7073165] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:29:45 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073165/ [7073166] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:22:45 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073166/ [7073167] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:34:41 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073167/ [7073168] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:20 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073168/ [7073169] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:49:13 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073169/ [7073170] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:52:59 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073170/ [7073171] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:53:06 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073171/ [7073172] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073172/ [7073173] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:21:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073173/ [7073174] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:58:24 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073174/ [7073175] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:26:47 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073175/ [7073177] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:20:51 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073177/ [7073178] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:12:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073178/ [7073180] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:21:49 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073180/ [7073181] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:19 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073181/ [7073182] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:24:16 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073182/ [7073183] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:47:16 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073183/ [7073184] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 02:08:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073184/ [7073185] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:37 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073185/ [7073186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:09:34 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073186/ [7073188] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073188/ [7073189] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:12:04 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073189/ [7073190] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:08 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073190/ [7073191] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:36 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073191/ [7073193] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:11:12 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073193/ [7073194] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 01:14:39 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073194/ [7073196] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:35:55 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073196/ [7073197] fs/verify/{begin/{0-install 1-ceph 2-logrotate} centos_8 clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:17:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073197/ [7073198] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:37:16 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073198/ [7073199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:12:40 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073199/ [7073200] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:22:57 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073200/ [7073201] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073201/ [7073202] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:15:56 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073202/ [7073203] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:24:56 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073203/ [7073204] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:35:10 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073204/ [7073205] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:42:27 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073205/ [7073206] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:33 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073206/ [7073207] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:16:20 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073207/ [7073208] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:38 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073208/ [7073210] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:28:32 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073210/ [7073211] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:20:56 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073211/ [7073212] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:11:48 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073212/ [7073213] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:36:32 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073213/ [7073214] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:27:40 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073214/ [7073215] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:05 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073215/ [7073216] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 01:12:41 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073216/ [7073217] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:48:26 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073217/ [7073218] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:32:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073218/ [7073219] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:54 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073219/ [7073220] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:09 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073220/ [7073222] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:20:57 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073222/ [7073223] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:17:46 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073223/ [7073224] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:10 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073224/ [7073225] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:38:13 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073225/ [7073226] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:32:26 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073226/ [7073228] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:13:43 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073228/ [7073229] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 00:48:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073229/ [7073230] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:31:33 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073230/ [7073231] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:56:41 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073231/ [7073233] fs/workload/{0-centos_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:39:01 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073233/ [7073234] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_latest k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:15:18 info: http://pulpito.ceph.com/vshankar-2022-10-18_18:05:47-fs-wip-vshankar-testing1-20221018-100814-testing-default-smithi/7073234/