2022-09-14T06:17:08.393 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-14T06:17:08.397 INFO:teuthology.results:subset: '83/256'
2022-09-14T06:17:08.397 INFO:teuthology.results:seed: '4029'
2022-09-14T07:56:27.005 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-14T07:56:27.033 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-14T19:37:25.927 INFO:teuthology.results:Tests finished! gathering results...
2022-09-14T19:37:34.347 INFO:scrape:Found 85 jobs
2022-09-14T19:37:50.570 INFO:scrape:Found 13 distinct failure reasons
2022-09-14T19:37:50.571 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a127ffb7df2a46d9c720771310f2b324a2af81f2
2022-09-14T19:37:50.572 INFO:scrape:1 jobs: ['7031670']
2022-09-14T19:37:50.572 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2022-09-14T19:37:50.573 INFO:scrape:
2022-09-14T19:37:50.573 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'
2022-09-14T19:37:50.574 INFO:scrape:1 jobs: ['7031633']
2022-09-14T19:37:50.575 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2022-09-14T19:37:50.576 INFO:scrape:
2022-09-14T19:37:50.577 INFO:scrape:Failure: Test failure: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)
2022-09-14T19:37:50.577 INFO:scrape:2 jobs: ['7031634', '7031700']
2022-09-14T19:37:50.578 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}']
2022-09-14T19:37:50.579 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/admin}']
2022-09-14T19:37:50.580 INFO:scrape:
2022-09-14T19:37:50.581 INFO:scrape:Dead: b'2022-09-14T19:23:11.407 DEBUG:teuthology.exit:Finished running handlers'
2022-09-14T19:37:50.582 INFO:scrape:1 jobs: ['7031676']
2022-09-14T19:37:50.583 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-09-14T19:37:50.583 INFO:scrape:
2022-09-14T19:37:50.584 INFO:scrape:Failure: "1663143551.3875659 mon.a (mon.0) 118 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-09-14T19:37:50.585 INFO:scrape:5 jobs: ['7031715', '7031714', '7031652', '7031709', '7031667']
2022-09-14T19:37:50.585 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2022-09-14T19:37:50.586 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/5', 'rhel_8}', 'session_timeout}', 'tasks/asok_dump_tree}', 'tasks/backtrace}', 'tasks/dbench', 'tasks/metrics}', 'tasks/snap-schedule}', 'validater/lockdep}']
2022-09-14T19:37:50.587 INFO:scrape:
2022-09-14T19:37:50.587 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-09-14T19:37:50.588 INFO:scrape:2 jobs: ['7031665', '7031632']
2022-09-14T19:37:50.588 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2022-09-14T19:37:50.589 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2022-09-14T19:37:50.589 INFO:scrape:
2022-09-14T19:37:50.590 INFO:scrape:Failure: "1663138686.6717978 mds.a (mds.2) 1 : cluster [WRN] client.4821 isn't responding to mclientcaps(revoke), ino 0x30000000d07 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.006127 seconds ago" in cluster log
2022-09-14T19:37:50.590 INFO:scrape:1 jobs: ['7031636']
2022-09-14T19:37:50.591 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-09-14T19:37:50.592 INFO:scrape:
2022-09-14T19:37:50.593 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
2022-09-14T19:37:50.593 INFO:scrape:2 jobs: ['7031645', '7031678']
2022-09-14T19:37:50.594 INFO:scrape:suites intersection: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-09-14T19:37:50.595 INFO:scrape:suites union: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-09-14T19:37:50.596 INFO:scrape:
2022-09-14T19:37:50.597 INFO:scrape:Failure: Command failed on smithi007 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45c81b8a-3401-11ed-8431-001a4aab830c -- bash -c 'ceph fs dump'"
2022-09-14T19:37:50.598 INFO:scrape:1 jobs: ['7031685']
2022-09-14T19:37:50.598 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2022-09-14T19:37:50.599 INFO:scrape:
2022-09-14T19:37:50.600 INFO:scrape:Failure: Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2022-09-14T19:37:50.600 INFO:scrape:1 jobs: ['7031653']
2022-09-14T19:37:50.601 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2022-09-14T19:37:50.601 INFO:scrape:
2022-09-14T19:37:50.602 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2022-09-14T19:37:50.602 INFO:scrape:1 jobs: ['7031647']
2022-09-14T19:37:50.603 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/no}']
2022-09-14T19:37:50.603 INFO:scrape:
2022-09-14T19:37:50.604 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-09-14T19:37:50.604 INFO:scrape:1 jobs: ['7031666']
2022-09-14T19:37:50.605 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-09-14T19:37:50.605 INFO:scrape:
2022-09-14T19:37:50.606 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi192 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2022-09-14T19:37:50.606 INFO:scrape:1 jobs: ['7031637']
2022-09-14T19:37:50.607 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2022-09-14T19:37:50.607 INFO:scrape:
2022-09-14T19:37:50.612 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/
failed:  19
dead:    1
running: 0
waiting: 0
queued:  0
passed:  65


Fail
=================================================================
[7031632]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   01:04:00
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031632/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031632/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ac37623643a245fe9b5d74bd4645b043

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7031633]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:20:44
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031633/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031633/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb247025565c41e6a337e8c9a6bcfe48

    Command failed (workunit test fs/quota/quota.sh) on smithi033 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd --
    /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3
    CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'

[7031634]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:24:23
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031634/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031634/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8555feb39a0542f392c8532122319352

    Test failure: test_dump_loads
    (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)

[7031636]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:36:27
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031636/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031636/

    "1663138686.6717978 mds.a (mds.2) 1 : cluster [WRN] client.4821 isn't
    responding to mclientcaps(revoke), ino 0x30000000d07 pending pAsLsXsFscr
    issued pAsLsXsFsxcrwb, sent 300.006127 seconds ago" in cluster log

[7031637]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:27:34
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031637/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031637/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cc6e23dfebc147f8a77c87737d04678b

    Command failed (workunit test kernel_untar_build.sh) on smithi192 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7031645]  fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
-----------------------------------------------------------------
time:   00:19:44
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031645/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031645/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef70e960e2324389b2f6edf4142f380f

    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)

[7031647]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:32:23
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031647/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031647/
sentry: https://sentry.ceph.com/organizations/ceph/?query=486fdc2f55e944c18c81e21c0f2da13e

    Command failed (workunit test suites/pjd.sh) on smithi093 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b40ecbf0136791c92ac1badb59c4772694a1940d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7031652]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:41:07
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031652/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031652/

    "1663139871.205943 mon.a (mon.0) 1642 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7031653]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:42:21
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031653/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031653/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fa1c50feff2343069ea21c61da7fb162

    Test failure: test_subvolume_group_ls_filter_internal_directories
    (tasks.cephfs.test_volumes.TestSubvolumeGroups)

[7031665]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   01:07:14
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031665/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031665/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02725189c9984ace851c341551ffc967

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7031666]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:21:28
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031666/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031666/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8c1111465ca841678bfef5c228bc91ed

    reached maximum tries (90) after waiting for 540 seconds

[7031667]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:48:37
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031667/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031667/

    "1663139971.9966989 mon.a (mon.0) 490 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7031670]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031670/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031670/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4ab2c5960b2428e9c0773fe3f669e7e

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh
    a1=a127ffb7df2a46d9c720771310f2b324a2af81f2

[7031678]  fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031678/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031678/
sentry: https://sentry.ceph.com/organizations/ceph/?query=421eed2770124e78806778b86b806e18

    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)

[7031685]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:24:43
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031685/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031685/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b67ff14b66964b45b49e17943f481391

    Command failed on smithi007 with status 127: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/daemon-base:latest-
    pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
    --fsid 45c81b8a-3401-11ed-8431-001a4aab830c -- bash -c 'ceph fs dump'"

[7031700]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:17:46
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031700/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031700/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30eb93bb3105420ea5cc6db79da4eb72

    Test failure: test_dump_loads
    (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)

[7031709]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:31:58
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031709/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031709/

    "1663142987.9211397 mon.a (mon.0) 220 : cluster [WRN] Health check failed:
    Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded
    (PG_DEGRADED)" in cluster log

[7031714]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031714/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031714/

    "1663143639.4423876 mon.a (mon.0) 239 : cluster [WRN] Health check failed:
    Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded
    (PG_DEGRADED)" in cluster log

[7031715]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:54:19
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031715/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031715/

    "1663143551.3875659 mon.a (mon.0) 118 : cluster [WRN] Health check failed:
    Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded
    (PG_DEGRADED)" in cluster log



Dead
=================================================================
[7031676]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031676/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031676/

    hit max job timeout



Pass
=================================================================
[7031635] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:19:29
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031635/

[7031638] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:55:21
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031638/

[7031639] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:41:23
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031639/

[7031640] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:29:55
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031640/

[7031641] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:55
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031641/

[7031642] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iogen}}
time:   01:10:52
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031642/

[7031643] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:27:32
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031643/

[7031644] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:34
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031644/

[7031646] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:44:16
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031646/

[7031648] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:46:26
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031648/

[7031649] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:33
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031649/

[7031650] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:32:25
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031650/

[7031651] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:39:57
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031651/

[7031654] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:28:41
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031654/

[7031655] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:44:52
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031655/

[7031656] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:21:25
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031656/

[7031657] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/ffsb}}
time:   01:31:10
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031657/

[7031658] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:46
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031658/

[7031659] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:28:27
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031659/

[7031660] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:21:46
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031660/

[7031661] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:05:42
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031661/

[7031662] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:00:30
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031662/

[7031663] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:20
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031663/

[7031664] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:33:03
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031664/

[7031668] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:37
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031668/

[7031669] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:15:16
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031669/

[7031671] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:54:50
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031671/

[7031672] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:16
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031672/

[7031673] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/norstats}}
time:   00:27:25
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031673/

[7031674] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:23
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031674/

[7031675] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:38:42
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031675/

[7031677] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:31:16
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031677/

[7031679] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:34:58
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031679/

[7031680] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:58:34
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031680/

[7031681] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
time:   02:10:19
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031681/

[7031682] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:16:29
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031682/

[7031683] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:25:04
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031683/

[7031684] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:22:14
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031684/

[7031686] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:32:17
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031686/

[7031687] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:14:52
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031687/

[7031688] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsx}}
time:   00:30:26
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031688/

[7031689] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:11
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031689/

[7031690] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:15:59
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031690/

[7031691] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:27:17
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031691/

[7031692] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:41:01
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031692/

[7031693] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:21:42
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031693/

[7031694] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:20:50
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031694/

[7031695] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:21:29
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031695/

[7031696] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:20:56
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031696/

[7031697] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:42:01
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031697/

[7031698] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:17:48
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031698/

[7031699] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:38:13
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031699/

[7031701] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:16:35
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031701/

[7031702] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:55:12
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031702/

[7031703] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:30:40
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031703/

[7031704] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:15:25
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031704/

[7031705] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:25:04
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031705/

[7031706] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:43
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031706/

[7031707] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:14:42
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031707/

[7031708] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:41
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031708/

[7031710] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:35:10
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031710/

[7031711] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:42:41
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031711/

[7031712] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:22:48
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031712/

[7031713] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:13:41
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031713/

[7031716] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:35:05
info:   http://pulpito.ceph.com/vshankar-2022-09-14_06:15:24-fs-wip-vshankar-testing1-20220905-132828-testing-default-smithi/7031716/