2023-09-08T07:03:31.074 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-08T07:03:31.082 INFO:teuthology.results:subset: '45/1024'
2023-09-08T07:03:31.082 INFO:teuthology.results:seed: '8546'
2023-09-08T07:04:28.179 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-08T07:04:28.180 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-08T19:05:15.994 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-09-08T19:05:20.096 INFO:scrape:Found 35 jobs
2023-09-08T19:05:21.275 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391032/teuthology.log
2023-09-08T19:05:25.431 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.432 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.432 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.433 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.437 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.437 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.438 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:25.438 WARNING:scrape:Saw end of BT but not start
2023-09-08T19:05:33.514 INFO:scrape:Found 13 distinct failure reasons
2023-09-08T19:05:33.516 INFO:scrape:Failure: Command failed on smithi136 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-09-08T19:05:33.516 INFO:scrape:4 jobs: ['7391029', '7391042', '7391013', '7391027']
2023-09-08T19:05:33.517 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-09-08T19:05:33.517 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-09-08T19:05:33.518 INFO:scrape:
2023-09-08T19:05:33.518 INFO:scrape:Dead: b'2023-09-08T19:05:11.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2023-09-08T19:05:33.519 INFO:scrape:4 jobs: ['7391028', '7391014', '7391036', '7391016']
2023-09-08T19:05:33.519 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-09-08T19:05:33.520 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}', 'tasks/xfstests-dev}']
2023-09-08T19:05:33.520 INFO:scrape:
2023-09-08T19:05:33.520 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-09-08T19:05:33.521 INFO:scrape:1 jobs: ['7391044']
2023-09-08T19:05:33.521 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-09-08T19:05:33.522 INFO:scrape:
2023-09-08T19:05:33.522 INFO:scrape:Dead: None
2023-09-08T19:05:33.523 INFO:scrape:1 jobs: ['7391032']
2023-09-08T19:05:33.523 INFO:scrape:
2023-09-08T19:05:33.523 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'
2023-09-08T19:05:33.524 INFO:scrape:2 jobs: ['7391045', '7391017']
2023-09-08T19:05:33.524 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-09-08T19:05:33.525 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-09-08T19:05:33.525 INFO:scrape:
2023-09-08T19:05:33.525 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-09-08T19:05:33.526 INFO:scrape:1 jobs: ['7391015']
2023-09-08T19:05:33.526 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-09-08T19:05:33.527 INFO:scrape:
2023-09-08T19:05:33.527 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-09-08T19:05:33.527 INFO:scrape:1 jobs: ['7391043']
2023-09-08T19:05:33.528 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-09-08T19:05:33.528 INFO:scrape:
2023-09-08T19:05:33.529 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi153 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-09-08T19:05:33.529 INFO:scrape:2 jobs: ['7391033', '7391025']
2023-09-08T19:05:33.529 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-09-08T19:05:33.530 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-09-08T19:05:33.530 INFO:scrape:
2023-09-08T19:05:33.531 INFO:scrape:Dead: b'2023-09-08T19:05:24.293 INFO:journalctl@ceph.mon.b.smithi105.stdout:Sep 08 19:05:23 smithi105 ceph-mon[100284]: pgmap v22093: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s'
2023-09-08T19:05:33.531 INFO:scrape:2 jobs: ['7391047', '7391046']
2023-09-08T19:05:33.531 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-09-08T19:05:33.532 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/yes', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/blogbench}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fail_fs/yes', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2023-09-08T19:05:33.532 INFO:scrape:
2023-09-08T19:05:33.532 INFO:scrape:Failure: "2023-09-08T07:31:08.078703+0000 mon.a (mon.0) 123 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-09-08T19:05:33.533 INFO:scrape:2 jobs: ['7391019', '7391022']
2023-09-08T19:05:33.533 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
2023-09-08T19:05:33.534 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}']
2023-09-08T19:05:33.534 INFO:scrape:
2023-09-08T19:05:33.534 INFO:scrape:Failure: "2023-09-08T07:37:56.609540+0000 osd.6 (osd.6) 3 : cluster [WRN] OSD bench result of 136088.206206 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.6. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-09-08T19:05:33.535 INFO:scrape:1 jobs: ['7391031']
2023-09-08T19:05:33.535 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-09-08T19:05:33.536 INFO:scrape:
2023-09-08T19:05:33.536 INFO:scrape:Dead: b'2023-09-08T19:05:30.638 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 08 19:05:30 smithi203 ceph-mon[99939]: pgmap v21690: 97 pgs: 97 active+clean; 8.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s'
2023-09-08T19:05:33.536 INFO:scrape:1 jobs: ['7391018']
2023-09-08T19:05:33.537 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2023-09-08T19:05:33.537 INFO:scrape:
2023-09-08T19:05:33.538 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-09-08T19:05:33.538 INFO:scrape:1 jobs: ['7391034']
2023-09-08T19:05:33.538 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-09-08T19:05:33.539 INFO:scrape:
2023-09-08T19:05:33.541 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/
failed:  15
dead:    1
running: 7
waiting: 0
queued:  0
passed:  12


Fail
=================================================================
[7391013]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:17:26
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391013/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391013/
sentry: https://sentry.ceph.com/organizations/ceph/?query=38adfca30ca3454b86283a572cd30852

    Command failed on smithi114 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7391015]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:53:59
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391015/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391015/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7391017]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:18:58
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391017/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391017/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e35b31d21be4d0c932306895681d72a

    Command failed (workunit test fs/quota/quota.sh) on smithi188 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh'

[7391019]  fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
-----------------------------------------------------------------
time:   02:10:49
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391019/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391019/

    "2023-09-08T07:31:08.078703+0000 mon.a (mon.0) 123 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7391022]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:28:43
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391022/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391022/

    "2023-09-08T07:34:16.124395+0000 mon.a (mon.0) 97 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7391025]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:25:58
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391025/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391025/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b2b19edb3384efeacb893cdd4cccd27

    Command failed (workunit test kernel_untar_build.sh) on smithi164 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7391027]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:16:43
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391027/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391027/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5394f2ba5a5548feb82174e83a92aa84

    Command failed on smithi044 with status 8: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7391029]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391029/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391029/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f86ac520b49c4f41a7164cf89d6aa5ac

    Command failed on smithi136 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7391031]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:27:01
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391031/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391031/

    "2023-09-08T07:37:56.609540+0000 osd.6 (osd.6) 3 : cluster [WRN] OSD bench
    result of 136088.206206 IOPS exceeded the threshold limit of 500.000000
    IOPS for osd.6. IOPS capacity is unchanged at 315.000000 IOPS. The
    recommendation is to establish the osd's IOPS capacity using other
    benchmark tools (e.g. Fio) and then override
    osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log

[7391033]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:27:17
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391033/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391033/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4db156505ddb4b00a8381146f694f79b

    Command failed (workunit test kernel_untar_build.sh) on smithi153 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7391034]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:17:11
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391034/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391034/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7eff57257de24e4ebde946e3e2eb586a

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7391042]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:19:00
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391042/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391042/
sentry: https://sentry.ceph.com/organizations/ceph/?query=988b671ff466483e9f095ac2823bfeaa

    Command failed on smithi118 with status 8: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7391043]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   08:23:12
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391043/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391043/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01593136f2cd407c8670de8daf567631

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7391044]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:18:04
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391044/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391044/
sentry: https://sentry.ceph.com/organizations/ceph/?query=59b799456e7540f2a715a2160fad6014

    Command failed (workunit test fs/test_python.sh) on smithi053 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7391045]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:19:12
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391045/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391045/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0ad81fe29e74d109a094bea6da7da26

    Command failed (workunit test fs/quota/quota.sh) on smithi040 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'



Dead
=================================================================
[7391032]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391032/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391032/

    Error reimaging machines: Timeout opening channel.



Running
=================================================================
[7391014] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391014/

[7391016] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391016/

[7391018] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391018/

[7391028] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391028/

[7391036] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391036/

[7391046] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391046/

[7391047] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391047/



Pass
=================================================================
[7391020] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:22:50
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391020/

[7391021] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:18:52
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391021/

[7391023] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:15
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391023/

[7391024] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:51:20
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391024/

[7391026] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:34
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391026/

[7391030] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:22:16
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391030/

[7391035] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:04:42
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391035/

[7391037] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   02:10:17
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391037/

[7391038] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:53
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391038/

[7391039] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:23:13
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391039/

[7391040] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
time:   00:32:48
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391040/

[7391041] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:16:06
info:   http://pulpito.ceph.com/vshankar-2023-09-08_07:03:01-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7391041/