2023-05-10T02:35:02.341 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-10T02:35:02.344 INFO:teuthology.results:subset: '572/1024'
2023-05-10T02:35:02.344 INFO:teuthology.results:seed: '2501'
2023-05-10T06:38:36.603 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-10T06:38:36.648 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-10T18:38:41.042 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-05-10T18:38:58.442 INFO:scrape:Found 206 jobs
2023-05-10T18:39:22.496 INFO:scrape:Found 13 distinct failure reasons
2023-05-10T18:39:22.497 INFO:scrape:Failure: Command failed on smithi016 with status 22: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 ''"
2023-05-10T18:39:22.498 INFO:scrape:19 jobs: ['7269180', '7269147', '7269203', '7269292', '7269263', '7269131', '7269230', '7269265', '7269323', '7269316', '7269235', '7269157', '7269277', '7269221', '7269126', '7269255', '7269164', '7269188', '7269169']
2023-05-10T18:39:22.498 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-05-10T18:39:22.499 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2023-05-10T18:39:22.499 INFO:scrape:
2023-05-10T18:39:22.499 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-05-10T18:39:22.500 INFO:scrape:2 jobs: ['7269145', '7269319']
2023-05-10T18:39:22.500 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-05-10T18:39:22.500 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-05-10T18:39:22.501 INFO:scrape:
2023-05-10T18:39:22.501 INFO:scrape:Failure: Command failed on smithi141 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 96c9c9a4-eef3-11ed-9b02-001a4aab830c -e sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"
2023-05-10T18:39:22.501 INFO:scrape:11 jobs: ['7269240', '7269222', '7269128', '7269276', '7269163', '7269183', '7269259', '7269148', '7269289', '7269202', '7269321']
2023-05-10T18:39:22.502 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2023-05-10T18:39:22.502 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4']
2023-05-10T18:39:22.502 INFO:scrape:
2023-05-10T18:39:22.503 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-05-10T18:39:22.503 INFO:scrape:2 jobs: ['7269201', '7269181']
2023-05-10T18:39:22.503 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-05-10T18:39:22.504 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-05-10T18:39:22.504 INFO:scrape:
2023-05-10T18:39:22.504 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-05-10T18:39:22.505 INFO:scrape:1 jobs: ['7269248']
2023-05-10T18:39:22.505 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-05-10T18:39:22.505 INFO:scrape:
2023-05-10T18:39:22.506 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-05-10T18:39:22.506 INFO:scrape:2 jobs: ['7269318', '7269154']
2023-05-10T18:39:22.506 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-05-10T18:39:22.506 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-05-10T18:39:22.507 INFO:scrape:
2023-05-10T18:39:22.507 INFO:scrape:Dead: b'2023-05-10T17:19:03.095 DEBUG:teuthology.exit:Finished running handlers'
2023-05-10T18:39:22.507 INFO:scrape:3 jobs: ['7269243', '7269297', '7269212']
2023-05-10T18:39:22.508 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}']
2023-05-10T18:39:22.508 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}', 'wsync/yes}']
2023-05-10T18:39:22.508 INFO:scrape:
2023-05-10T18:39:22.509 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-05-10T18:39:22.509 INFO:scrape:1 jobs: ['7269257']
2023-05-10T18:39:22.509 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-05-10T18:39:22.509 INFO:scrape:
2023-05-10T18:39:22.510 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench'
2023-05-10T18:39:22.510 INFO:scrape:3 jobs: ['7269141', '7269179', '7269219']
2023-05-10T18:39:22.510 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}']
2023-05-10T18:39:22.511 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}']
2023-05-10T18:39:22.511 INFO:scrape:
2023-05-10T18:39:22.511 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-05-10T18:39:22.512 INFO:scrape:1 jobs: ['7269261']
2023-05-10T18:39:22.512 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}']
2023-05-10T18:39:22.512 INFO:scrape:
2023-05-10T18:39:22.512 INFO:scrape:Failure: Test failure: test_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)
2023-05-10T18:39:22.513 INFO:scrape:1 jobs: ['7269159']
2023-05-10T18:39:22.513 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}']
2023-05-10T18:39:22.513 INFO:scrape:
2023-05-10T18:39:22.514 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'
2023-05-10T18:39:22.514 INFO:scrape:2 jobs: ['7269322', '7269152']
2023-05-10T18:39:22.514 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-05-10T18:39:22.514 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}']
2023-05-10T18:39:22.515 INFO:scrape:
2023-05-10T18:39:22.515 INFO:scrape:Failure: "1683693788.8697934 mon.a (mon.0) 378 : cluster [WRN] Health check failed: Degraded data redundancy: 15/2050 objects degraded (0.732%), 1 pg degraded (PG_DEGRADED)" in cluster log
2023-05-10T18:39:22.515 INFO:scrape:1 jobs: ['7269198']
2023-05-10T18:39:22.516 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2023-05-10T18:39:22.516 INFO:scrape:
2023-05-10T18:39:22.530 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/
failed:  46
dead:    2
running: 1
waiting: 0
queued:  0
passed:  157


Fail
=================================================================
[7269126]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:23:59
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269126/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269126/
sentry: https://sentry.ceph.com/organizations/ceph/?query=443f635db5c84f12a3156465a79622d9

    Command failed on smithi005 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269128]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:36
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269128/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269128/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f88439fdbd51400fbc1d519c21b1bdf5

    Command failed on smithi017 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5588c8d6-eee0-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269131]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:18:40
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269131/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269131/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fb6f8ee108494613a59616fe8e612ffb

    Command failed on smithi158 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7269141]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
-----------------------------------------------------------------
time:   00:22:29
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269141/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269141/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a7918bbb35240a59f3455f782c6a461

    Command failed (workunit test fs/fscrypt.sh) on smithi181 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    dbench'

[7269145]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   09:21:01
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269145/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=815dd5e1c78e494cbc87b92f0ff0617b

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7269147]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:32:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269147/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=651accf69a5e4224a9565251fa76f5b4

    Command failed on smithi070 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269148]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:20:47
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269148/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70eccf053cb343c1ba4292dcd66471d7

    Command failed on smithi082 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1ddd0dc-
    eee2-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"

[7269152]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:22:54
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269152/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269152/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e4772b3385d4252b7aa280b400baf61

    Command failed (workunit test fs/quota/quota.sh) on smithi073 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh'

[7269154]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   06:58:29
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269154/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269154/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae0e305f4ed149c0a18ba8185df36c78

    Command failed (workunit test suites/fsstress.sh) on smithi023 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7269157]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:27:04
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269157/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=43b59a4786404f50a256896e0b6f1162

    Command failed on smithi044 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269159]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:43:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269159/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269159/
sentry: https://sentry.ceph.com/organizations/ceph/?query=136d64063cfe470b852b867fa93ef522

    Test failure: test_single_path_rootsquash
    (tasks.cephfs.test_admin.TestFsAuthorize)

[7269163]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269163/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269163/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a1066bf4b40046eb9806ee4be7f41497

    Command failed on smithi132 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    9de3b0ec-eee4-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269164]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:16:39
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269164/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269164/
sentry: https://sentry.ceph.com/organizations/ceph/?query=094738a9d33f4a13b4e758e04d4e7c2a

    Command failed on smithi131 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7269169]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:28:35
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269169/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae713edd89a14581a170b8020bf12d2e

    Command failed on smithi017 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269179]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
-----------------------------------------------------------------
time:   00:14:46
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269179/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269179/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7412206060d54c448aa4e47553bb0c11

    Command failed (workunit test fs/fscrypt.sh) on smithi196 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb'

[7269180]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:27:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269180/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269180/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33c8199320984685b67a734cdbf4cff1

    Command failed on smithi016 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269181]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   01:11:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269181/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269181/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3fd74baa07064a779e46bcc3f0715e4c

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi029 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7269183]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:21:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269183/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269183/
sentry: https://sentry.ceph.com/organizations/ceph/?query=04c1eaf81926411393c12f04c1075555

    Command failed on smithi096 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    209000f2-eee7-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269188]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:26:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269188/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269188/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f60767ff0e5482c961031c49675da4a

    Command failed on smithi032 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269198]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:59:53
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269198/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269198/

    "1683693788.8697934 mon.a (mon.0) 378 : cluster [WRN] Health check failed:
    Degraded data redundancy: 15/2050 objects degraded (0.732%), 1 pg degraded
    (PG_DEGRADED)" in cluster log

[7269201]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:50:55
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269201/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269201/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a51026258b044ca183918c9ac9a33b77

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi017 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7269202]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269202/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269202/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c154df39c0f74ef7a007c14bb7c45825

    Command failed on smithi112 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    0f2a6112-eeed-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"

[7269203]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:29:25
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269203/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269203/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ebf14705fc84303816301f1c9300a26

    Command failed on smithi007 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269219]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
-----------------------------------------------------------------
time:   00:15:50
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269219/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269219/
sentry: https://sentry.ceph.com/organizations/ceph/?query=08036fbf8f7e4f8b9087fde7899b3db6

    Command failed (workunit test fs/fscrypt.sh) on smithi159 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    iozone'

[7269221]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:25:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269221/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269221/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1abf8b403d5b4e96a7f89e6aede985f7

    Command failed on smithi012 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269222]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:53
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269222/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269222/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5b4b58b48ce1456dadbd4ec39a761b3c

    Command failed on smithi107 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1d24f77a-eef0-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269230]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:16:59
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269230/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269230/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0440a2d6b2464eb6a9ac976d555a9c53

    Command failed on smithi156 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7269235]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:28:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269235/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269235/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb0432179004487eba5627bc51946f65

    Command failed on smithi066 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269240]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:23:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269240/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269240/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f13145d110a4617ab03b632bc62e8df

    Command failed on smithi141 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    96c9c9a4-eef3-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"

[7269248]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:16:54
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269248/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269248/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9d0f4cebb0d43f5a1048ddfe1c07d1a

    Command failed (workunit test fs/test_python.sh) on smithi003 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7269255]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:28:14
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269255/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269255/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57890659fe624d2d98e92948325fc2b4

    Command failed on smithi002 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269257]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:18:45
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269257/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269257/
sentry: https://sentry.ceph.com/organizations/ceph/?query=46bf05d451f3447fb92a06e5b612aed1

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[7269259]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:21:11
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269259/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269259/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b6a27dc0592e4d7997c0fd36e41afa67

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3b0face8-eef6-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269261]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:13:51
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269261/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269261/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f8853ac265f94ee8a42748617ddb3881

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7269263]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:25:11
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269263/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269263/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9822f09849254d379cb0f1a1197a6bc4

    Command failed on smithi035 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269265]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:17:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269265/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269265/
sentry: https://sentry.ceph.com/organizations/ceph/?query=932b9bfe72844b6b989024b177b191dc

    Command failed on smithi158 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7269276]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:19:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269276/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269276/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3f2ab13f0f2b4c118e31163d08b9baaf

    Command failed on smithi131 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b6733c08-eef9-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"

[7269277]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:26:30
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269277/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269277/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f945a819258b4dc381e1b4ac4d935bc2

    Command failed on smithi107 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269289]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:20:49
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269289/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269289/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3cb76c8d130f456081e1f9976dd1840e

    Command failed on smithi082 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b769c530-eefb-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7269292]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:21:59
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269292/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269292/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e7861068e334a26bf44df57cfe73aa2

    Command failed on smithi032 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269316]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:29:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269316/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269316/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4716f432e6c54a9a974f0e16fe156dee

    Command failed on smithi063 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"

[7269318]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   06:51:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269318/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269318/
sentry: https://sentry.ceph.com/organizations/ceph/?query=590a761f3e26464e9846665ccade6e99

    Command failed (workunit test suites/fsstress.sh) on smithi002 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7269319]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   10:28:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269319/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269319/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e3cb5f9b68d7443c8de5f20d1053b0a1

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7269321]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:19:58
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269321/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269321/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cbf873ccfb8e42748fca90a5438c7d8c

    Command failed on smithi032 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7c588838-eeff-11ed-9b02-001a4aab830c -e
    sha1=ce38c2c7ab99c152279feeefe476d4b72227c734 -- bash -c 'ceph orch ps'"

[7269322]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:25:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269322/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269322/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3aac07087cef4da59f62fa9c78fda9aa

    Command failed (workunit test fs/quota/quota.sh) on smithi053 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ce38c2c7ab99c152279feeefe476d4b72227c734
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'

[7269323]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:28:32
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269323/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269323/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a557b846af3d47aa8e719e3770c99a0d

    Command failed on smithi043 with status 22: "sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs subvolume create cephfs sv_0 ''"



Dead
=================================================================
[7269212]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269212/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269212/

    hit max job timeout

[7269297]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269297/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269297/

    hit max job timeout



Running
=================================================================
[7269243] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269243/



Pass
=================================================================
[7269118] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:13:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269118/

[7269119] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:25:02
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269119/

[7269120] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:59:09
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269120/

[7269121] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:38:30
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269121/

[7269122] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:38:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269122/

[7269123] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:51
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269123/

[7269124] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269124/

[7269125] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:58:04
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269125/

[7269127] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:38:44
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269127/

[7269129] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:25:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269129/

[7269130] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:20:59
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269130/

[7269132] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:58:14
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269132/

[7269133] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:09:53
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269133/

[7269134] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:36:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269134/

[7269135] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:18:34
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269135/

[7269136] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:24:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269136/

[7269137] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:38:28
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269137/

[7269138] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:58:12
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269138/

[7269139] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:22:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269139/

[7269140] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:36:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269140/

[7269142] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269142/

[7269143] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:22:49
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269143/

[7269144] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/emptystring}}
time:   00:21:56
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269144/

[7269146] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:20:31
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269146/

[7269149] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269149/

[7269150] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:16:07
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269150/

[7269151] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:20:32
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269151/

[7269153] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:31:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269153/

[7269155] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:17:21
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269155/

[7269156] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:39:55
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269156/

[7269158] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:26:41
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269158/

[7269160] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:44:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269160/

[7269161] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:14
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269161/

[7269162] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:13:57
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269162/

[7269165] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:30:19
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269165/

[7269166] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:57:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269166/

[7269167] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:48
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269167/

[7269168] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:26:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269168/

[7269170] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:22:14
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269170/

[7269171] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:24:28
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269171/

[7269172] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:15:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269172/

[7269173] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:00
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269173/

[7269174] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:49
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269174/

[7269175] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:09:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269175/

[7269176] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:56:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269176/

[7269177] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:33:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269177/

[7269178] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:14:12
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269178/

[7269182] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:17:30
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269182/

[7269184] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:02:09
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269184/

[7269185] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:21:25
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269185/

[7269186] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:25:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269186/

[7269187] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:21:56
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269187/

[7269189] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:27:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269189/

[7269190] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:13:04
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269190/

[7269191] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   00:52:07
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269191/

[7269192] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
time:   00:49:53
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269192/

[7269193] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:38:55
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269193/

[7269194] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:24:53
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269194/

[7269195] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:02
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269195/

[7269196] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:20:33
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269196/

[7269197] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:23:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269197/

[7269199] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269199/

[7269200] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:44
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269200/

[7269204] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:30:37
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269204/

[7269205] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:11
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269205/

[7269206] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:16
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269206/

[7269207] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:38:48
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269207/

[7269208] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:35
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269208/

[7269209] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:12:09
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269209/

[7269210] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:47:19
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269210/

[7269211] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:45:06
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269211/

[7269213] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:22:47
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269213/

[7269214] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:50:34
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269214/

[7269215] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   01:02:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269215/

[7269216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:20:12
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269216/

[7269217] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:14:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269217/

[7269218] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:54
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269218/

[7269220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:20:44
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269220/

[7269223] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:35:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269223/

[7269224] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:36
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269224/

[7269225] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:16:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269225/

[7269226] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:23:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269226/

[7269227] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:13
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269227/

[7269228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:17:25
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269228/

[7269229] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:30:18
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269229/

[7269231] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:27:11
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269231/

[7269232] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:52:28
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269232/

[7269233] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:45:08
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269233/

[7269234] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:51:58
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269234/

[7269236] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:25:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269236/

[7269237] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:20:59
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269237/

[7269238] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:51
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269238/

[7269239] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:17:52
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269239/

[7269241] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:21:35
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269241/

[7269242] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:36:47
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269242/

[7269244] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:59:51
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269244/

[7269245] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:19:57
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269245/

[7269246] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:24:35
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269246/

[7269247] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:38
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269247/

[7269249] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:46
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269249/

[7269250] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:00
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269250/

[7269251] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:41:03
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269251/

[7269252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:19:13
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269252/

[7269253] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:49:27
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269253/

[7269254] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269254/

[7269256] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:11:22
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269256/

[7269258] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:24:24
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269258/

[7269260] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   00:23:27
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269260/

[7269262] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:20:32
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269262/

[7269264] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:23:12
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269264/

[7269266] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:54:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269266/

[7269267] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:30:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269267/

[7269268] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:29:10
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269268/

[7269269] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:36:49
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269269/

[7269270] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:31:19
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269270/

[7269271] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   00:57:02
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269271/

[7269272] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:22:40
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269272/

[7269273] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:49
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269273/

[7269274] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:48
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269274/

[7269275] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269275/

[7269278] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/emptystring}}
time:   00:21:04
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269278/

[7269279] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:50:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269279/

[7269280] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:33
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269280/

[7269281] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:14:45
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269281/

[7269282] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:22:40
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269282/

[7269283] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:37:20
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269283/

[7269284] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:07
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269284/

[7269285] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:28:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269285/

[7269286] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:52:06
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269286/

[7269287] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:24:04
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269287/

[7269288] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:36
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269288/

[7269290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:29:35
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269290/

[7269291] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:21:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269291/

[7269293] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:25:34
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269293/

[7269294] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:25:44
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269294/

[7269295] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:14:20
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269295/

[7269296] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:47:21
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269296/

[7269298] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:25:06
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269298/

[7269299] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:11:55
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269299/

[7269300] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_20.04} tasks/mirror}
time:   02:14:39
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269300/

[7269301] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   00:30:12
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269301/

[7269302] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:00:45
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269302/

[7269303] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:39:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269303/

[7269304] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:21:26
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269304/

[7269305] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:49:16
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269305/

[7269306] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:19:23
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269306/

[7269307] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:16:36
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269307/

[7269308] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:14:43
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269308/

[7269309] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:19:41
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269309/

[7269310] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:18
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269310/

[7269311] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:28:29
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269311/

[7269312] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:15:18
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269312/

[7269313] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:10:01
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269313/

[7269314] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:51:42
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269314/

[7269315] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:26:15
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269315/

[7269317] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:14:47
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269317/

[7269320] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:24:50
info:   http://pulpito.ceph.com/vshankar-2023-05-10_02:31:38-fs-wip-vshankar-testing-20230509.090957-testing-default-smithi/7269320/