2023-12-12T08:47:09.439 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-12T08:47:09.440 INFO:teuthology.results:subset: '600/1024'
2023-12-12T08:47:09.440 INFO:teuthology.results:seed: '5473'
2023-12-12T12:28:19.455 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-12T12:28:19.490 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-12T21:59:08.096 INFO:teuthology.results:Tests finished! gathering results...
2023-12-12T21:59:26.079 INFO:scrape:Found 203 jobs
2023-12-12T21:59:52.574 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488292/teuthology.log
2023-12-12T21:59:54.092 INFO:scrape:Found 15 distinct failure reasons
2023-12-12T21:59:54.092 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-12-12T21:59:54.092 INFO:scrape:1 jobs: ['7488184']
2023-12-12T21:59:54.092 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-12-12T21:59:54.093 INFO:scrape:
2023-12-12T21:59:54.093 INFO:scrape:Failure: Test failure: test_fscrypt_encrypt (tasks.cephfs.test_fscrypt.TestFscrypt)
2023-12-12T21:59:54.093 INFO:scrape:1 jobs: ['7488264']
2023-12-12T21:59:54.093 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client']
2023-12-12T21:59:54.093 INFO:scrape:
2023-12-12T21:59:54.093 INFO:scrape:Failure: valgrind error: Leak_StillReachable
calloc
calloc
_dl_check_map_versions
2023-12-12T21:59:54.093 INFO:scrape:3 jobs: ['7488196', '7488276', '7488137']
2023-12-12T21:59:54.093 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
2023-12-12T21:59:54.094 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{ubuntu/{overrides', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'tasks/mirror}}', 'ubuntu_latest}}', 'validater/valgrind}']
2023-12-12T21:59:54.094 INFO:scrape:
2023-12-12T21:59:54.094 INFO:scrape:Failure: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh'
2023-12-12T21:59:54.094 INFO:scrape:1 jobs: ['7488220']
2023-12-12T21:59:54.094 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2023-12-12T21:59:54.094 INFO:scrape:
2023-12-12T21:59:54.094 INFO:scrape:Failure: Command failed on smithi097 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 pull'
2023-12-12T21:59:54.094 INFO:scrape:2 jobs: ['7488335', '7488230']
2023-12-12T21:59:54.095 INFO:scrape:suites intersection: []
2023-12-12T21:59:54.095 INFO:scrape:suites union: ['1-rename', 'distro/single-container-host', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}']
2023-12-12T21:59:54.095 INFO:scrape:
2023-12-12T21:59:54.095 INFO:scrape:Failure: Command failed on smithi156 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6dfba232-98cf-11ee-95a3-87774f69a715 -e sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2023-12-12T21:59:54.095 INFO:scrape:10 jobs: ['7488177', '7488211', '7488321', '7488250', '7488226', '7488271', '7488303', '7488155', '7488285', '7488194']
2023-12-12T21:59:54.095 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2023-12-12T21:59:54.095 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2023-12-12T21:59:54.096 INFO:scrape:
2023-12-12T21:59:54.096 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/dbench.sh
2023-12-12T21:59:54.096 INFO:scrape:1 jobs: ['7488239']
2023-12-12T21:59:54.096 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/lockdep}']
2023-12-12T21:59:54.096 INFO:scrape:
2023-12-12T21:59:54.096 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-12-12T21:59:54.096 INFO:scrape:1 jobs: ['7488236']
2023-12-12T21:59:54.096 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}']
2023-12-12T21:59:54.096 INFO:scrape:
2023-12-12T21:59:54.097 INFO:scrape:Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
2023-12-12T21:59:54.097 INFO:scrape:1 jobs: ['7488201']
2023-12-12T21:59:54.097 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-12-12T21:59:54.097 INFO:scrape:
2023-12-12T21:59:54.097 INFO:scrape:Failure: No module named 'tasks.fs'
2023-12-12T21:59:54.097 INFO:scrape:1 jobs: ['7488309']
2023-12-12T21:59:54.097 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}']
2023-12-12T21:59:54.097 INFO:scrape:
2023-12-12T21:59:54.097 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)
2023-12-12T21:59:54.097 INFO:scrape:1 jobs: ['7488252']
2023-12-12T21:59:54.098 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/admin}']
2023-12-12T21:59:54.098 INFO:scrape:
2023-12-12T21:59:54.098 INFO:scrape:Failure: Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays)
2023-12-12T21:59:54.098 INFO:scrape:1 jobs: ['7488219']
2023-12-12T21:59:54.098 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']
2023-12-12T21:59:54.098 INFO:scrape:
2023-12-12T21:59:54.098 INFO:scrape:Dead: b'2023-12-12T20:55:54.111 DEBUG:teuthology.exit:Finished running handlers'
2023-12-12T21:59:54.098 INFO:scrape:1 jobs: ['7488157']
2023-12-12T21:59:54.098 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/lockdep}']
2023-12-12T21:59:54.099 INFO:scrape:
2023-12-12T21:59:54.099 INFO:scrape:Dead: None
2023-12-12T21:59:54.099 INFO:scrape:1 jobs: ['7488292']
2023-12-12T21:59:54.099 INFO:scrape:
2023-12-12T21:59:54.099 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-12-12T21:59:54.099 INFO:scrape:1 jobs: ['7488242']
2023-12-12T21:59:54.099 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-12-12T21:59:54.099 INFO:scrape:
2023-12-12T21:59:54.110 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/
failed:  25
dead:    2
running: 0
waiting: 0
queued:  0
passed:  176


Fail
=================================================================
[7488137]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:32:03
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488137/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488137/

    valgrind error: Leak_StillReachable malloc malloc strdup

[7488155]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:20:23
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488155/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488155/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5690308231b44ea992b52b9b2dd7ed88

    Command failed on smithi062 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    eca722aa-98cf-11ee-95a3-87774f69a715 -- ceph mon dump -f json'

[7488177]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:41:19
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488177/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488177/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3abdbb582a8945dc825b996ea377ad3d

    Command failed on smithi156 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6dfba232-98cf-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488184]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:20:50
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488184/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488184/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e341174a6034b089f9db369b3338899

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[7488194]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:57
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488194/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488194/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42baac0f679347878785a4038ba32c80

    Command failed on smithi163 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    11dae52a-98d0-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488196]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:35:27
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488196/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488196/

    valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions

[7488201]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:18:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488201/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488201/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35a61c0c9f444b6fba08fb2a253f9da2

    Test failure: test_r_with_fsname_and_no_path_in_cap
    (tasks.cephfs.test_multifs_auth.TestMDSCaps)

[7488211]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:34:49
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488211/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488211/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6430e58aaa4c4946b80feabfbb5463b5

    Command failed on smithi099 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    eee23f4c-98d3-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488219]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   00:17:16
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488219/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488219/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bdeee2297b444d0396a66fc8d3cf51cd

    Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays)

[7488220]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:39:32
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488220/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488220/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b02660a13c2240ecbf2796147cb87b96

    Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi042 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/misc/general_vxattrs.sh'

[7488226]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:37:49
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488226/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488226/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24d0e94d8f204282a727025a2bc9b5d0

    Command failed on smithi017 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0eaa0ea8-98d5-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488230]  fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
-----------------------------------------------------------------
time:   00:24:49
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488230/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488230/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0bd16a980954eb18430f833ad561c0b

    Command failed on smithi049 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    68fc49ec-98d6-11ee-95a3-87774f69a715 -- bash -c 'ceph fs volume rename foo
    bar --yes-i-really-mean-it'"

[7488236]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:54:22
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488236/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488236/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7488239]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   03:29:33
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488239/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488239/
sentry: https://sentry.ceph.com/organizations/ceph/?query=224be16339da4cda89cd0f51bb5ce8a6

    Command failed (workunit test suites/dbench.sh) on smithi087 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7488242]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   09:21:18
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488242/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488242/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4ad4348061cf44aebb891b9964ad44b5

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7488250]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:36:19
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488250/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488250/
sentry: https://sentry.ceph.com/organizations/ceph/?query=73df5484e8174b0aa613d2b7c1e02f6f

    Command failed on smithi007 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f371b3a4-98d7-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488252]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:39:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488252/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488252/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be27498520df46a2a350b7abad72cf2b

    Test failure: test_multiple_path_r
    (tasks.cephfs.test_admin.TestFsAuthorize)

[7488264]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}}
-----------------------------------------------------------------
time:   02:33:14
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488264/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488264/
sentry: https://sentry.ceph.com/organizations/ceph/?query=23599a5d385c47d8b2e5568914173a1a

    Test failure: test_fscrypt_encrypt (tasks.cephfs.test_fscrypt.TestFscrypt)

[7488271]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:36:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488271/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488271/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07099f27868845eeb96dbeb0e3a3da51

    Command failed on smithi032 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c93552be-98da-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488276]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:35:27
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488276/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488276/

    valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions

[7488285]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:35:23
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488285/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488285/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff990d3c08d34f4f87d491753742ea65

    Command failed on smithi063 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    85e33c70-98e3-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488303]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:35:36
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488303/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488303/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e4e8c33f8364796ab257d79d73cbde7

    Command failed on smithi152 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    52fac614-98e5-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488309]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:24:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488309/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488309/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3eeb565cbbce4b61bed7380325d7c665

    No module named 'tasks.fs'

[7488321]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488321/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488321/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e15466c33974468b0a826e7ff6f78b2

    Command failed on smithi038 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f36c7b22-98ec-11ee-95a3-87774f69a715 -e
    sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7488335]  fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:11:37
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488335/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488335/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b83c33a60894e92a544e3eec390a0cb

    Command failed on smithi097 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 pull'



Dead
=================================================================
[7488157]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488157/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488157/

    hit max job timeout

[7488292]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488292/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488292/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Pass
=================================================================
[7488138] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:57:37
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488138/

[7488139] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:49:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488139/

[7488140] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:12
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488140/

[7488141] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:23:16
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488141/

[7488142] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
time:   00:31:51
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488142/

[7488143] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:59:37
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488143/

[7488144] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:19:19
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488144/

[7488145] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:26:14
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488145/

[7488146] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:23:10
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488146/

[7488147] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:19:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488147/

[7488148] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:23:49
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488148/

[7488149] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:18:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488149/

[7488150] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}}
time:   00:44:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488150/

[7488151] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:43:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488151/

[7488152] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:29:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488152/

[7488153] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:25:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488153/

[7488154] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:04:56
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488154/

[7488156] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:20:04
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488156/

[7488158] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:54
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488158/

[7488159] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488159/

[7488160] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:28:30
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488160/

[7488161] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:23:04
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488161/

[7488162] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:04:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488162/

[7488163] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:31:31
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488163/

[7488164] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:27:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488164/

[7488165] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:31:41
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488165/

[7488166] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:18:52
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488166/

[7488167] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:20:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488167/

[7488168] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:20:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488168/

[7488169] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:53:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488169/

[7488170] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:24:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488170/

[7488171] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:47:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488171/

[7488172] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:24:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488172/

[7488173] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:37:04
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488173/

[7488174] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:31
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488174/

[7488175] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:24:00
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488175/

[7488176] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:19:56
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488176/

[7488178] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:20:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488178/

[7488179] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
time:   00:55:39
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488179/

[7488180] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   00:56:05
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488180/

[7488181] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:42
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488181/

[7488182] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:18:55
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488182/

[7488183] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:17
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488183/

[7488185] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}}
time:   01:57:54
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488185/

[7488186] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:18:21
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488186/

[7488187] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488187/

[7488188] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:50:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488188/

[7488189] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488189/

[7488190] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:19:32
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488190/

[7488191] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:35
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488191/

[7488192] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}}
time:   00:44:06
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488192/

[7488193] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:50:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488193/

[7488195] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:16:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488195/

[7488197] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:22:33
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488197/

[7488198] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:56:54
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488198/

[7488199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:31:59
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488199/

[7488200] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:28:08
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488200/

[7488202] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:44:57
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488202/

[7488203] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
time:   00:28:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488203/

[7488204] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:06
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488204/

[7488205] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:18:03
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488205/

[7488206] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:42:56
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488206/

[7488207] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   01:51:50
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488207/

[7488208] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488208/

[7488209] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:11
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488209/

[7488210] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:17:28
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488210/

[7488212] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:15:51
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488212/

[7488213] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:26:05
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488213/

[7488214] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}}
time:   00:38:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488214/

[7488215] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:19:23
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488215/

[7488216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:28:07
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488216/

[7488217] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:23
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488217/

[7488218] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:28
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488218/

[7488221] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:41:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488221/

[7488222] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:18:11
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488222/

[7488223] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   00:54:28
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488223/

[7488224] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:01:52
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488224/

[7488225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:26:12
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488225/

[7488227] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:15:51
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488227/

[7488228] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:21:38
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488228/

[7488229] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:08
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488229/

[7488231] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:34:14
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488231/

[7488232] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:02:46
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488232/

[7488233] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:19:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488233/

[7488234] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:28
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488234/

[7488235] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:48:29
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488235/

[7488237] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:18:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488237/

[7488238] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:31:40
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488238/

[7488240] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488240/

[7488241] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:35:12
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488241/

[7488243] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:40
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488243/

[7488244] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   00:58:36
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488244/

[7488245] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:16:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488245/

[7488246] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
time:   00:52:22
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488246/

[7488247] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}}
time:   00:15:19
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488247/

[7488248] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:34:17
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488248/

[7488249] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:16:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488249/

[7488251] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:19:52
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488251/

[7488253] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:22:32
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488253/

[7488254] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}}
time:   00:42:05
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488254/

[7488255] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:31
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488255/

[7488256] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:42
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488256/

[7488257] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:15:49
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488257/

[7488258] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:19:43
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488258/

[7488259] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:12
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488259/

[7488260] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:47:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488260/

[7488261] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:19:50
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488261/

[7488262] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:19:08
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488262/

[7488263] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:18:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488263/

[7488265] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:16:10
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488265/

[7488266] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:27:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488266/

[7488267] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:37:06
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488267/

[7488268] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:52:34
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488268/

[7488269] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
time:   01:04:46
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488269/

[7488270] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:16:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488270/

[7488272] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:19:40
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488272/

[7488273] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:21
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488273/

[7488274] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:15:00
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488274/

[7488275] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   01:08:23
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488275/

[7488277] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}}
time:   00:43:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488277/

[7488278] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:15:09
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488278/

[7488279] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:19:42
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488279/

[7488280] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:27:17
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488280/

[7488281] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:23:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488281/

[7488282] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:36
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488282/

[7488283] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:16:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488283/

[7488284] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:33
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488284/

[7488286] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:21:22
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488286/

[7488287] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:15:16
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488287/

[7488288] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}}
time:   00:36:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488288/

[7488289] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488289/

[7488290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:55:28
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488290/

[7488291] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:19:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488291/

[7488293] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:28:54
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488293/

[7488294] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:45:01
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488294/

[7488295] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}}
time:   00:36:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488295/

[7488296] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:42:38
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488296/

[7488297] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:23:55
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488297/

[7488298] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:18:42
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488298/

[7488299] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:15:50
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488299/

[7488300] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:56:11
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488300/

[7488301] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   02:01:30
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488301/

[7488302] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   00:51:13
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488302/

[7488304] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:51:25
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488304/

[7488305] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488305/

[7488306] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:52
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488306/

[7488307] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:21:55
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488307/

[7488308] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:44:02
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488308/

[7488310] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:20:48
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488310/

[7488311] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488311/

[7488312] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:23:32
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488312/

[7488313] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
time:   00:53:26
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488313/

[7488314] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:21:36
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488314/

[7488315] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:19:04
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488315/

[7488316] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:35
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488316/

[7488317] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:33:06
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488317/

[7488318] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:27:12
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488318/

[7488319] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:16:51
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488319/

[7488320] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:20:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488320/

[7488322] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:26:47
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488322/

[7488323] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:37:05
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488323/

[7488324] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:17:14
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488324/

[7488325] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:17:22
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488325/

[7488326] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:24
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488326/

[7488327] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:16:39
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488327/

[7488328] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:51:40
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488328/

[7488329] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:25:53
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488329/

[7488330] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:16:32
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488330/

[7488331] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror}
time:   02:19:44
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488331/

[7488332] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit}
time:   00:40:29
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488332/

[7488333] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:01:58
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488333/

[7488334] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:40:20
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488334/

[7488336] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:40
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488336/

[7488337] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:50:56
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488337/

[7488338] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:41:07
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488338/

[7488339] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop}
time:   00:18:41
info:   https://pulpito.ceph.com/mchangir-2023-12-12_08:46:52-fs-wip-mchangir-testing-20231205.064351-testing-default-smithi/7488339/