2022-10-04T12:27:09.684 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-04T12:27:09.686 INFO:teuthology.results:subset: '74/256' 2022-10-04T12:27:09.686 INFO:teuthology.results:seed: '6295' 2022-10-04T19:25:16.864 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-04T19:25:16.888 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-05T03:52:03.172 INFO:teuthology.results:Tests finished! gathering results... 2022-10-05T03:52:27.433 INFO:scrape:Found 241 jobs 2022-10-05T03:53:46.633 INFO:scrape:Found 24 distinct failure reasons 2022-10-05T03:53:46.635 INFO:scrape:Failure: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2022-10-05T03:53:46.636 INFO:scrape:2 jobs: ['7053999', '7053798'] 2022-10-05T03:53:46.636 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}'] 2022-10-05T03:53:46.637 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/strays}'] 2022-10-05T03:53:46.637 INFO:scrape: 2022-10-05T03:53:46.638 INFO:scrape:Failure: Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 574d6f6a-441c-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2022-10-05T03:53:46.639 INFO:scrape:10 jobs: ['7054032', '7053916', '7053952', '7053935', '7053819', '7053853', '7053899', '7053806', '7054012', '7053976'] 2022-10-05T03:53:46.640 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2022-10-05T03:53:46.640 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2022-10-05T03:53:46.641 INFO:scrape: 2022-10-05T03:53:46.641 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2022-10-05T03:53:46.642 INFO:scrape:11 jobs: ['7053797', '7053920', '7053879', '7054018', '7053849', '7053818', '7053939', '7053889', '7053826', '7053841', '7053968'] 2022-10-05T03:53:46.643 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-10-05T03:53:46.643 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2022-10-05T03:53:46.644 INFO:scrape: 2022-10-05T03:53:46.645 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-10-05T03:53:46.645 INFO:scrape:5 jobs: ['7054005', '7053913', '7053854', '7053960', '7053811'] 2022-10-05T03:53:46.646 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay'] 2022-10-05T03:53:46.647 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-10-05T03:53:46.647 INFO:scrape: 2022-10-05T03:53:46.648 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2022-10-05T03:53:46.648 INFO:scrape:1 jobs: ['7053884'] 2022-10-05T03:53:46.649 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2022-10-05T03:53:46.649 INFO:scrape: 2022-10-05T03:53:46.650 INFO:scrape:Crash: Test failure: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-10-05T03:53:46.650 INFO:scrape:ceph version 18.0.0-273-g494a9a69 (494a9a6965bc84e443c45d7809c3b082513a1fac) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb558161ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x7fb5591a44f5] 5: (interval_set::insert(inodeno_t, inodeno_t, inodeno_t*, inodeno_t*)+0x46e) [0x562585d247ee] 6: (InoTable::replay_release_ids(interval_set&)+0x6d) [0x562585f401bd] 7: (ESession::replay(MDSRank*)+0xc1) [0x56258601d461] 8: (MDLog::_replay_thread()+0x9b0) [0x562585fb2860] 9: (MDLog::ReplayThread::entry()+0x11) [0x562585c6a7a1] 10: /lib64/libpthread.so.0(+0x81cf) [0x7fb5581571cf] 11: clone() 2022-10-05T03:53:46.651 INFO:scrape:1 jobs: ['7053903'] 2022-10-05T03:53:46.651 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}'] 2022-10-05T03:53:46.652 INFO:scrape: 2022-10-05T03:53:46.652 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1664907851.156:193): avc: denied { node_bind } for pid=1442 comm="ping" saddr=172.21.15.40 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-10-05T03:53:46.653 INFO:scrape:1 jobs: ['7053992'] 2022-10-05T03:53:46.653 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2022-10-05T03:53:46.654 INFO:scrape: 2022-10-05T03:53:46.655 INFO:scrape:Failure: Command failed (workunit test fs/misc/acl.sh) on smithi093 with status 6: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' 2022-10-05T03:53:46.655 INFO:scrape:8 jobs: ['7053929', '7054011', '7053866', '7053874', '7053964', '7053834', '7054015', '7053815'] 2022-10-05T03:53:46.656 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2022-10-05T03:53:46.657 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/permission/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_misc}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2022-10-05T03:53:46.657 INFO:scrape: 2022-10-05T03:53:46.658 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2022-10-05T03:53:46.658 INFO:scrape:1 jobs: ['7053892'] 2022-10-05T03:53:46.659 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-10-05T03:53:46.659 INFO:scrape: 2022-10-05T03:53:46.660 INFO:scrape:Failure: "1664899698.397398 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log 2022-10-05T03:53:46.660 INFO:scrape:12 jobs: ['7053907', '7053937', '7053958', '7053978', '7053868', '7053865', '7053965', '7053822', '7054025', '7053946', '7053940', '7053908'] 2022-10-05T03:53:46.661 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2022-10-05T03:53:46.661 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'test/clone}}', 'thrashosds-health}'] 2022-10-05T03:53:46.662 INFO:scrape: 2022-10-05T03:53:46.662 INFO:scrape:Failure: Command failed on smithi096 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:494a9a6965bc84e443c45d7809c3b082513a1fac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e35cab88-43f4-11ed-8433-001a4aab830c -- ceph orch host add smithi109' 2022-10-05T03:53:46.663 INFO:scrape:1 jobs: ['7053859'] 2022-10-05T03:53:46.663 INFO:scrape:suites: ['1-mount', '2-workload/dbench', 'distro/single-container-host}', 'fs/cephadm/multivolume/{0-start'] 2022-10-05T03:53:46.664 INFO:scrape: 2022-10-05T03:53:46.664 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2022-10-05T03:53:46.664 INFO:scrape:1 jobs: ['7053888'] 2022-10-05T03:53:46.665 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/damage}'] 2022-10-05T03:53:46.665 INFO:scrape: 2022-10-05T03:53:46.666 INFO:scrape:Failure: Kernel mounts did not umount cleanly 2022-10-05T03:53:46.666 INFO:scrape:1 jobs: ['7053830'] 2022-10-05T03:53:46.667 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-10-05T03:53:46.667 INFO:scrape: 2022-10-05T03:53:46.667 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi027 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2022-10-05T03:53:46.668 INFO:scrape:3 jobs: ['7053876', '7053880', '7054009'] 2022-10-05T03:53:46.668 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2022-10-05T03:53:46.669 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-10-05T03:53:46.669 INFO:scrape: 2022-10-05T03:53:46.670 INFO:scrape:Failure: Test failure: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads) 2022-10-05T03:53:46.670 INFO:scrape:2 jobs: ['7054031', '7053832'] 2022-10-05T03:53:46.671 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}'] 2022-10-05T03:53:46.671 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/admin}'] 2022-10-05T03:53:46.672 INFO:scrape: 2022-10-05T03:53:46.672 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) 2022-10-05T03:53:46.673 INFO:scrape:2 jobs: ['7053890', '7054004'] 2022-10-05T03:53:46.673 INFO:scrape:suites intersection: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus'] 2022-10-05T03:53:46.674 INFO:scrape:suites union: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus'] 2022-10-05T03:53:46.674 INFO:scrape: 2022-10-05T03:53:46.674 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-10-05T03:53:46.675 INFO:scrape:1 jobs: ['7053988'] 2022-10-05T03:53:46.676 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2022-10-05T03:53:46.676 INFO:scrape: 2022-10-05T03:53:46.676 INFO:scrape:Failure: Command failed on smithi155 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b69213c-43ed-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" 2022-10-05T03:53:46.677 INFO:scrape:1 jobs: ['7053840'] 2022-10-05T03:53:46.677 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2022-10-05T03:53:46.678 INFO:scrape: 2022-10-05T03:53:46.678 INFO:scrape:Failure: Test failure: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2022-10-05T03:53:46.679 INFO:scrape:1 jobs: ['7053970'] 2022-10-05T03:53:46.679 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/scrub}'] 2022-10-05T03:53:46.680 INFO:scrape: 2022-10-05T03:53:46.680 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2022-10-05T03:53:46.680 INFO:scrape:1 jobs: ['7053887'] 2022-10-05T03:53:46.681 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2022-10-05T03:53:46.681 INFO:scrape: 2022-10-05T03:53:46.682 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2022-10-05T03:53:46.682 INFO:scrape:1 jobs: ['7053893'] 2022-10-05T03:53:46.683 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2022-10-05T03:53:46.683 INFO:scrape: 2022-10-05T03:53:46.684 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) 2022-10-05T03:53:46.684 INFO:scrape:1 jobs: ['7053919'] 2022-10-05T03:53:46.684 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-flush}'] 2022-10-05T03:53:46.685 INFO:scrape: 2022-10-05T03:53:46.685 INFO:scrape:Dead: b'2022-10-05T03:49:03.326 DEBUG:teuthology.exit:Finished running handlers' 2022-10-05T03:53:46.686 INFO:scrape:1 jobs: ['7053904'] 2022-10-05T03:53:46.686 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2022-10-05T03:53:46.687 INFO:scrape: 2022-10-05T03:53:46.687 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-10-05T03:53:46.688 INFO:scrape:1 jobs: ['7053870'] 2022-10-05T03:53:46.688 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-10-05T03:53:46.688 INFO:scrape: 2022-10-05T03:53:46.703 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/ failed: 69 dead: 1 running: 0 waiting: 0 queued: 0 passed: 171 Fail ================================================================= [7053797] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:36:33 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053797/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053797/ error during scrub thrashing: rank damage found: {'backtrace'} [7053798] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053798/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63de8a506c8148e6b3bf5ef99177bcfe Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) [7053806] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053806/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dd734b7ea994dcc847b88aee4977c5d Command failed on smithi027 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1d0b9056-43e6-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7053811] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:50:47 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053811/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e4e15129b4349d2a868c2ea42f339d6 Command failed (workunit test suites/pjd.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7053815] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:32:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053815/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a89b3801807a404d824ee0efdc91f837 Command failed (workunit test fs/misc/acl.sh) on smithi079 with status 10: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053818] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:23:49 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053818/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053818/ error during scrub thrashing: rank damage found: {'backtrace'} [7053819] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053819/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d321c754b161487ebc2a2f54b838ed24 Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99bac2a0-43e8-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7053822] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:17:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053822/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053822/ "1664890876.1392627 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053826] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:51:39 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053826/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053826/ error during scrub thrashing: rank damage found: {'backtrace'} [7053830] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 02:01:49 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053830/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053830/ Kernel mounts did not umount cleanly [7053832] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:17:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053832/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e771fbd6fd9c4600baba0590c8b7e685 Test failure: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads) [7053834] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:13:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053834/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3f0fe99716743e8b9ef76944e4451fe Command failed (workunit test fs/misc/acl.sh) on smithi085 with status 6: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053840] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053840/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=799d88c6424a476daf25af5421c0030a Command failed on smithi155 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b69213c-43ed-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7053841] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:37:15 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053841/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053841/ error during scrub thrashing: rank damage found: {'backtrace'} [7053849] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 01:12:41 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053849/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053849/ error during scrub thrashing: rank damage found: {'backtrace'} [7053853] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:35:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053853/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69e7e7619969452d8336dbc93356746a Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2de9a828-43f4-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7053854] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:34:41 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053854/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=435560bda33049ccb782e198923ee11c Command failed (workunit test suites/pjd.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7053859] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:11:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053859/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8618616434da471dad475be17b645f36 Command failed on smithi096 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:494a9a6965bc84e443c45d7809c3b082513a1fac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e35cab88-43f4-11ed-8433-001a4aab830c -- ceph orch host add smithi109' [7053865] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:46:48 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053865/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053865/ "1664896546.227577 mon.a (mon.0) 738 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7053866] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:12:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053866/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053866/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28b2cf80354745e585b927613575b9c4 Command failed (workunit test fs/misc/acl.sh) on smithi008 with status 10: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053868] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:05:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053868/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053868/ "1664896551.8924115 mon.a (mon.0) 301 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053870] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:33:19 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053870/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053870/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bf18666fd94448088443155c9f865c8 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7053874] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:27:01 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053874/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053874/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76aa37e412e84ec0ba536d485db91248 Command failed (workunit test fs/misc/acl.sh) on smithi055 with status 10: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053876] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:45:38 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053876/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef7872e40bbd45388ef8a163c7ab100b Command failed (workunit test kernel_untar_build.sh) on smithi027 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7053879] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:46:13 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053879/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053879/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7053880] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053880/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d047f60d8dd54e2fb7fb276637caf252 Command failed (workunit test kernel_untar_build.sh) on smithi087 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7053884] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:28:08 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053884/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6907c53ba00845138fc08955b85f1c21 Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) [7053887] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:15:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053887/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053887/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16095afb627343d2adf8ec8cb3b087fd Command failed (workunit test suites/ffsb.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7053888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:13:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053888/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cddea67d0cf946c08344bd05304a98dc Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [7053889] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:27:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053889/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053889/ error during scrub thrashing: rank damage found: {'backtrace'} [7053890] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:13:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053890/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053890/ sentry: https://sentry.ceph.com/organizations/ceph/?query=313c313b6aa64fd0b791125ffa746463 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) [7053892] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:04:21 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053892/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053892/ Exiting scrub checking -- not all pgs scrubbed. [7053893] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:20:01 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053893/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b1ffb8e09b0417ca9ef4e5facc7bf7d Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [7053899] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:44 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053899/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c23c842c6e07400e8bef8ad8c679c7ad Command failed on smithi102 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1c150ee-43fc-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7053903] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:16:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053903/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c655f49d8cc47dd8e99605118328215 Test failure: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7053907] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:54:35 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053907/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053907/ "1664899698.397398 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053908] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:57:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053908/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053908/ "1664900685.6807683 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log [7053913] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:42:52 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053913/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df0d69ca3884417ba681292500800f66 Command failed (workunit test suites/pjd.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7053916] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053916/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66c3e10ab68148a794d0f7c6b56137e0 Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b919ef38-43ff-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7053919] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:12:57 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053919/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053919/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0cd71312a744dec9d210e1bd2f8b570 Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) [7053920] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:48:29 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053920/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053920/ error during scrub thrashing: rank damage found: {'backtrace'} [7053929] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:13:01 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053929/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=653bdb59db05477391c5b99b15730e39 Command failed (workunit test fs/misc/acl.sh) on smithi093 with status 6: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053935] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053935/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc593a3da7734a8b9f636b2764f2a8d5 Command failed on smithi027 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fc36cd82-4403-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7053937] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:32:03 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053937/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053937/ "1664902535.1905656 mon.a (mon.0) 1623 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7053939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:27:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053939/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053939/ error during scrub thrashing: rank damage found: {'backtrace'} [7053940] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:57:10 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053940/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053940/ "1664902480.029308 mon.a (mon.0) 99 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053946] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:52:16 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053946/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053946/ "1664903319.03255 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053952] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:08 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053952/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82658a6261cd4fe6a50bd7af75446eef Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88db50d4-4407-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7053958] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:50:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053958/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053958/ "1664905627.4300847 mon.a (mon.0) 85 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053960] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:26:26 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053960/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053960/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c74a0f665b74e7e8808276697e5ba78 Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7053964] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:34:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053964/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053964/ sentry: https://sentry.ceph.com/organizations/ceph/?query=889e01786b984c04b06054a6dae95eef Command failed (workunit test fs/misc/acl.sh) on smithi040 with status 10: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7053965] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:43:15 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053965/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053965/ "1664906682.6561537 mon.a (mon.0) 311 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7053968] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:59:54 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053968/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053968/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7053970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:14:47 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053970/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27c4fd14d75146aa97e15f5d0cd38fcc Test failure: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) [7053976] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:22 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053976/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ca59d23557049b1a60f40340a15836e Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f9446ce-440f-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7053978] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:50:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053978/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053978/ "1664907054.812604 mon.a (mon.0) 439 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7053988] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:50:29 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053988/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0b2405fd6a447b69ebaeac06af6d42d Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7053992] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:35:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053992/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053992/ SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1664907851.156:193): avc: denied { node_bind } for pid=1442 comm="ping" saddr=172.21.15.40 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7053999] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:23:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053999/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053999/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51583bd8d01a4b7cb3a5b48ca8f23836 Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) [7054004] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:12:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054004/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=490945abb1514cc9bcaaa5a108953b7b Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) [7054005] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:55 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054005/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f94716196d74a838592b47489be178b Command failed (workunit test suites/pjd.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7054009] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:32:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054009/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054009/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3f2c894a6f44c8bace069d172fa7d98 Command failed (workunit test kernel_untar_build.sh) on smithi197 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7054011] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:34:13 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054011/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054011/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c08a61ae2124481ab0428dabbbd8587 Command failed (workunit test fs/misc/acl.sh) on smithi005 with status 10: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7054012] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:46 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054012/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054012/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd4ffaa0878b413aa15488f6fdb9a29c Command failed on smithi052 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2f08d22c-4419-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7054015] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:13:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054015/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054015/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40c373b4a44d4a3cba3a921eac602126 Command failed (workunit test fs/misc/acl.sh) on smithi080 with status 6: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=494a9a6965bc84e443c45d7809c3b082513a1fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh' [7054018] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:02:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054018/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054018/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7054025] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:38:25 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054025/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054025/ "1664912137.7484877 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7054031] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:24:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054031/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054031/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70750498e407449bb7f974deb3455f72 Test failure: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads) [7054032] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054032/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d9e41c5492e4c0fbf43cc2ad8537611 Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 574d6f6a-441c-11ed-8433-001a4aab830c -e sha1=494a9a6965bc84e443c45d7809c3b082513a1fac -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' Dead ================================================================= [7053904] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053904/ hit max job timeout Pass ================================================================= [7053794] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:38:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053794/ [7053795] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053795/ [7053796] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:06 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053796/ [7053799] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:38:46 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053799/ [7053800] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:36:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053800/ [7053801] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053801/ [7053802] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:12:19 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053802/ [7053803] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:26 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053803/ [7053804] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053804/ [7053805] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:26:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053805/ [7053807] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 01:04:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053807/ [7053808] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:35:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053808/ [7053809] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:20:27 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053809/ [7053810] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:06:04 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053810/ [7053812] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053812/ [7053813] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:14:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053813/ [7053814] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:34:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053814/ [7053816] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:55 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053816/ [7053817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:13:33 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053817/ [7053820] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:17:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053820/ [7053821] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:44 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053821/ [7053823] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:45:10 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053823/ [7053824] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 02:05:45 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053824/ [7053825] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:14:35 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053825/ [7053827] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:18:34 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053827/ [7053828] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:22:03 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053828/ [7053829] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:56:34 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053829/ [7053831] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:54 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053831/ [7053833] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:26:10 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053833/ [7053835] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 00:39:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053835/ [7053836] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:04 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053836/ [7053837] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:19:23 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053837/ [7053838] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:39:56 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053838/ [7053839] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:55:22 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053839/ [7053842] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:25:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053842/ [7053843] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:43:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053843/ [7053844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:53 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053844/ [7053845] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:35:02 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053845/ [7053846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:02 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053846/ [7053847] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:56 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053847/ [7053848] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:44:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053848/ [7053850] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:15:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053850/ [7053851] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:20:08 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053851/ [7053852] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:34:45 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053852/ [7053855] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:11:58 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053855/ [7053856] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053856/ [7053857] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:47 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053857/ [7053858] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:11:39 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053858/ [7053860] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:26:44 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053860/ [7053861] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:19:03 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053861/ [7053862] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} time: 02:12:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053862/ [7053863] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:37:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053863/ [7053864] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:51 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053864/ [7053867] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:48:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053867/ [7053869] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:11:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053869/ [7053871] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:13:45 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053871/ [7053872] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:44:56 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053872/ [7053873] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:21:06 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053873/ [7053875] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:08 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053875/ [7053877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053877/ [7053878] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:18:43 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053878/ [7053881] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:03:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053881/ [7053882] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:55:10 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053882/ [7053883] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:10:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053883/ [7053885] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:44:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053885/ [7053886] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:32:50 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053886/ [7053891] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} time: 01:23:29 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053891/ [7053894] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:47:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053894/ [7053895] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:32:21 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053895/ [7053896] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053896/ [7053897] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053897/ [7053898] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:52:23 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053898/ [7053900] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:42:07 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053900/ [7053901] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:20:02 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053901/ [7053902] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:16 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053902/ [7053905] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053905/ [7053906] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:55:19 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053906/ [7053909] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:23 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053909/ [7053910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:26:38 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053910/ [7053911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053911/ [7053912] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:25:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053912/ [7053914] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:19:41 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053914/ [7053915] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:25:35 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053915/ [7053917] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:24:27 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053917/ [7053918] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 02:39:19 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053918/ [7053921] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:36:16 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053921/ [7053922] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:01 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053922/ [7053923] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 02:03:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053923/ [7053924] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:24:31 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053924/ [7053925] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:12:45 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053925/ [7053926] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:24:00 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053926/ [7053927] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:24:55 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053927/ [7053928] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:19:21 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053928/ [7053930] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:25 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053930/ [7053931] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} time: 01:29:02 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053931/ [7053932] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:12:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053932/ [7053933] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:15:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053933/ [7053934] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:55:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053934/ [7053936] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:57:25 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053936/ [7053938] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:42:26 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053938/ [7053941] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:18:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053941/ [7053942] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:35:38 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053942/ [7053943] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:55:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053943/ [7053944] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:16:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053944/ [7053945] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053945/ [7053947] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:19 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053947/ [7053948] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:25:21 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053948/ [7053949] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:21:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053949/ [7053950] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:42:46 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053950/ [7053951] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:47 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053951/ [7053953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:36:39 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053953/ [7053954] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:13:35 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053954/ [7053955] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:45:12 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053955/ [7053956] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:14 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053956/ [7053957] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053957/ [7053959] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:21:13 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053959/ [7053961] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:25 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053961/ [7053962] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:32:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053962/ [7053963] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:19:31 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053963/ [7053966] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:21:43 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053966/ [7053967] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:20:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053967/ [7053969] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:40 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053969/ [7053971] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 01:35:04 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053971/ [7053972] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:47:43 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053972/ [7053973] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:30:53 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053973/ [7053974] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:15:58 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053974/ [7053975] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:02:09 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053975/ [7053977] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} time: 01:36:54 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053977/ [7053979] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 01:22:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053979/ [7053980] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:41 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053980/ [7053981] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:35:37 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053981/ [7053982] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:25 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053982/ [7053983] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:20:33 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053983/ [7053984] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:21:04 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053984/ [7053985] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:02:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053985/ [7053986] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:33 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053986/ [7053987] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:18 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053987/ [7053989] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:14 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053989/ [7053990] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:13:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053990/ [7053991] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:32 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053991/ [7053993] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053993/ [7053994] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:12:21 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053994/ [7053995] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:24:41 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053995/ [7053996] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:38:06 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053996/ [7053997] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:10:48 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053997/ [7053998] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:35:17 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7053998/ [7054000] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 01:04:23 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054000/ [7054001] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:23:14 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054001/ [7054002] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:57 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054002/ [7054003] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:37:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054003/ [7054006] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054006/ [7054007] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:11:55 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054007/ [7054008] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:34:46 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054008/ [7054010] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:52:53 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054010/ [7054013] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:16:29 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054013/ [7054014] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:04 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054014/ [7054016] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 01:26:33 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054016/ [7054017] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:57 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054017/ [7054019] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:14:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054019/ [7054020] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 00:44:43 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054020/ [7054021] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:11 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054021/ [7054022] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:15:59 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054022/ [7054023] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054023/ [7054024] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:26:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054024/ [7054026] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:42 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054026/ [7054027] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} time: 00:56:28 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054027/ [7054028] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:42:15 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054028/ [7054029] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 02:06:05 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054029/ [7054030] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 00:37:30 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054030/ [7054033] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:39:52 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054033/ [7054034] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:19:55 info: http://pulpito.ceph.com/vshankar-2022-10-04_12:26:55-fs-wip-vshankar-testing-20221004-132749-testing-default-smithi/7054034/