2023-04-24T06:34:01.260 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-24T06:34:01.262 INFO:teuthology.results:subset: '807/1024'
2023-04-24T06:34:01.262 INFO:teuthology.results:seed: '8902'
2023-04-24T10:17:44.664 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-24T10:17:44.685 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-24T22:17:46.833 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-04-24T22:17:52.877 INFO:scrape:Found 75 jobs
2023-04-24T22:18:09.026 WARNING:scrape:Misunderstood line: 
2023-04-24T22:18:24.479 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249397/teuthology.log
2023-04-24T22:18:28.433 INFO:scrape:Found 26 distinct failure reasons
2023-04-24T22:18:28.434 INFO:scrape:Failure: more than one file system available
2023-04-24T22:18:28.435 INFO:scrape:16 jobs: ['7249391', '7249345', '7249394', '7249341', '7249396', '7249373', '7249331', '7249377', '7249358', '7249352', '7249333', '7249350', '7249327', '7249385', '7249364', '7249369']
2023-04-24T22:18:28.435 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-24T22:18:28.435 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-24T22:18:28.436 INFO:scrape:
2023-04-24T22:18:28.436 INFO:scrape:Failure: "1682332311.635119 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 143157.685375 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-04-24T22:18:28.436 INFO:scrape:1 jobs: ['7249395']
2023-04-24T22:18:28.437 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-04-24T22:18:28.437 INFO:scrape:
2023-04-24T22:18:28.437 INFO:scrape:Failure: Test failure: test_repeated_boot (tasks.cephfs.test_failover.TestFailover)
2023-04-24T22:18:28.438 INFO:scrape:3 jobs: ['7249324', '7249376', '7249348']
2023-04-24T22:18:28.438 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}']
2023-04-24T22:18:28.438 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}']
2023-04-24T22:18:28.439 INFO:scrape:
2023-04-24T22:18:28.439 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/dbench.sh
2023-04-24T22:18:28.439 INFO:scrape:3 jobs: ['7249365', '7249334', '7249362']
2023-04-24T22:18:28.440 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}', 'session_timeout}']
2023-04-24T22:18:28.440 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'frag', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'ranks/5', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/dbench', 'tasks/{0-subvolume/{with-quota}', 'validater/lockdep}', 'wsync/no}']
2023-04-24T22:18:28.440 INFO:scrape:
2023-04-24T22:18:28.441 INFO:scrape:Failure: Command failed on smithi099 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-04-24T22:18:28.441 INFO:scrape:3 jobs: ['7249336', '7249360', '7249375']
2023-04-24T22:18:28.441 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-04-24T22:18:28.442 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-04-24T22:18:28.442 INFO:scrape:
2023-04-24T22:18:28.442 INFO:scrape:Failure: Command failed (workunit test suites/blogbench.sh) on smithi053 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'
2023-04-24T22:18:28.443 INFO:scrape:1 jobs: ['7249367']
2023-04-24T22:18:28.443 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/blogbench}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/pacific']
2023-04-24T22:18:28.443 INFO:scrape:
2023-04-24T22:18:28.443 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi042 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-04-24T22:18:28.444 INFO:scrape:1 jobs: ['7249329']
2023-04-24T22:18:28.444 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-04-24T22:18:28.444 INFO:scrape:
2023-04-24T22:18:28.445 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-04-24T22:18:28.445 INFO:scrape:2 jobs: ['7249371', '7249380']
2023-04-24T22:18:28.445 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'wsync/no}']
2023-04-24T22:18:28.446 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2023-04-24T22:18:28.446 INFO:scrape:
2023-04-24T22:18:28.446 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-04-24T22:18:28.447 INFO:scrape:2 jobs: ['7249338', '7249382']
2023-04-24T22:18:28.447 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-04-24T22:18:28.447 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/valgrind}']
2023-04-24T22:18:28.447 INFO:scrape:
2023-04-24T22:18:28.448 INFO:scrape:Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
2023-04-24T22:18:28.448 INFO:scrape:1 jobs: ['7249390']
2023-04-24T22:18:28.448 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-04-24T22:18:28.449 INFO:scrape:
2023-04-24T22:18:28.449 INFO:scrape:Dead: b'2023-04-24T19:01:17.231 DEBUG:teuthology.exit:Finished running handlers'
2023-04-24T22:18:28.449 INFO:scrape:5 jobs: ['7249332', '7249381', '7249353', '7249387', '7249359']
2023-04-24T22:18:28.450 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}']
2023-04-24T22:18:28.450 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/suites/ffsb}}', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'msgr-failures/osd-mds-delay', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'wsync/no}']
2023-04-24T22:18:28.450 INFO:scrape:
2023-04-24T22:18:28.450 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/dbench.sh
2023-04-24T22:18:28.451 INFO:scrape:3 jobs: ['7249340', '7249383', '7249366']
2023-04-24T22:18:28.451 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_dbench_iozone}']
2023-04-24T22:18:28.451 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_dbench_iozone}']
2023-04-24T22:18:28.452 INFO:scrape:
2023-04-24T22:18:28.452 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi023 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'
2023-04-24T22:18:28.452 INFO:scrape:1 jobs: ['7249347']
2023-04-24T22:18:28.452 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}']
2023-04-24T22:18:28.453 INFO:scrape:
2023-04-24T22:18:28.453 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi171 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2023-04-24T22:18:28.453 INFO:scrape:1 jobs: ['7249325']
2023-04-24T22:18:28.454 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/fsstress}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/pacific']
2023-04-24T22:18:28.454 INFO:scrape:
2023-04-24T22:18:28.454 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-04-24T22:18:28.455 INFO:scrape:1 jobs: ['7249351']
2023-04-24T22:18:28.455 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-04-24T22:18:28.455 INFO:scrape:
2023-04-24T22:18:28.455 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration), test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
2023-04-24T22:18:28.456 INFO:scrape:1 jobs: ['7249355']
2023-04-24T22:18:28.456 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}']
2023-04-24T22:18:28.456 INFO:scrape:
2023-04-24T22:18:28.456 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/fs/fscrypt.sh none dbench
2023-04-24T22:18:28.457 INFO:scrape:1 jobs: ['7249326']
2023-04-24T22:18:28.457 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}']
2023-04-24T22:18:28.457 INFO:scrape:
2023-04-24T22:18:28.458 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2023-04-24T22:18:28.458 INFO:scrape:1 jobs: ['7249388']
2023-04-24T22:18:28.458 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{rhel_8}', 'workloads/cephfs-mirror-ha-workunit}']
2023-04-24T22:18:28.458 INFO:scrape:
2023-04-24T22:18:28.459 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi188 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'
2023-04-24T22:18:28.459 INFO:scrape:1 jobs: ['7249335']
2023-04-24T22:18:28.459 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/kernel_untar_build}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus']
2023-04-24T22:18:28.460 INFO:scrape:
2023-04-24T22:18:28.460 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'
2023-04-24T22:18:28.460 INFO:scrape:1 jobs: ['7249363']
2023-04-24T22:18:28.460 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-04-24T22:18:28.461 INFO:scrape:
2023-04-24T22:18:28.461 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi078 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-04-24T22:18:28.461 INFO:scrape:ceph version 18.0.0-3547-gbafd1bf9 (bafd1bf961ae8b7842aac9cf2a9ed311270e6977) reef (dev)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f75e7812cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f75e8ca9dcd]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f39) [0x7f75e8ca9f39]
 6: (Client::add_update_cap(Inode*, MetaSession*, unsigned long, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, UserPerm const&)+0xc53) [0x564fda1c0b33]
 7: (Client::add_update_inode(InodeStat*, utime_t, MetaSession*, UserPerm const&)+0x943) [0x564fda1c1f03]
 8: (Client::insert_trace(MetaRequest*, MetaSession*)+0xd6e) [0x564fda1c537e]
 9: (Client::handle_client_reply(boost::intrusive_ptr<MClientReply const> const&)+0x3a0) [0x564fda1c6140]
 10: (Client::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x58d) [0x564fda1e944d]
 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f75e8f21c18]
 12: (DispatchQueue::entry()+0x50f) [0x7f75e8f1edbf]
 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75e8fe6bf1]
 14: /lib64/libpthread.so.0(+0x81ca) [0x7f75e78081ca]
 15: clone()
2023-04-24T22:18:28.462 INFO:scrape:1 jobs: ['7249398']
2023-04-24T22:18:28.462 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-04-24T22:18:28.462 INFO:scrape:
2023-04-24T22:18:28.462 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-04-24T22:18:28.463 INFO:scrape:1 jobs: ['7249346']
2023-04-24T22:18:28.463 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/dbench}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/pacific']
2023-04-24T22:18:28.463 INFO:scrape:
2023-04-24T22:18:28.464 INFO:scrape:Dead: None
2023-04-24T22:18:28.464 INFO:scrape:1 jobs: ['7249397']
2023-04-24T22:18:28.464 INFO:scrape:
2023-04-24T22:18:28.464 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-04-24T22:18:28.465 INFO:scrape:1 jobs: ['7249368']
2023-04-24T22:18:28.465 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-04-24T22:18:28.465 INFO:scrape:
2023-04-24T22:18:28.466 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-04-24T22:18:28.466 INFO:scrape:1 jobs: ['7249343']
2023-04-24T22:18:28.466 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-04-24T22:18:28.466 INFO:scrape:
2023-04-24T22:18:28.467 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/iozone.sh
2023-04-24T22:18:28.467 INFO:scrape:1 jobs: ['7249356']
2023-04-24T22:18:28.467 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/iozone}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus']
2023-04-24T22:18:28.467 INFO:scrape:
2023-04-24T22:18:28.472 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/
failed:  49
dead:    5
running: 1
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[7249324]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:33:11
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249324/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249324/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8b0d3fe987f4b4685242870cc943118

    Test failure: test_repeated_boot (tasks.cephfs.test_failover.TestFailover)

[7249325]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
-----------------------------------------------------------------
time:   01:48:03
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249325/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249325/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24e86c30e0c44008ad23a20f6982a26c

    Command failed (workunit test suites/fsstress.sh) on smithi171 with status
    3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7249326]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
-----------------------------------------------------------------
time:   06:15:45
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249326/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249326/
sentry: https://sentry.ceph.com/organizations/ceph/?query=805af4ea0e1e4003a57c86b66dcf43f6

    Command failed (workunit test fs/fscrypt.sh) on smithi162 with status 124:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    dbench'

[7249327]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:14:49
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249327/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249327/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71680f1989014ee5ab0f6f66f28f0b03

    more than one file system available

[7249329]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:22:53
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249329/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249329/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2f0ee2410ecb4adc813d66cb7a17f217

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi042
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7249331]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:23:49
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249331/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249331/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1ae2af7baff4165a723a0138ac7bac2

    more than one file system available

[7249333]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:16:55
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249333/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249333/
sentry: https://sentry.ceph.com/organizations/ceph/?query=989268e3d3f8461baeca07f48352ad8b

    more than one file system available

[7249334]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   03:15:57
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249334/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249334/
sentry: https://sentry.ceph.com/organizations/ceph/?query=46e7714a9a5142a38b12fcba404d476f

    Command failed (workunit test suites/dbench.sh) on smithi143 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7249335]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
-----------------------------------------------------------------
time:   01:46:08
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249335/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249335/
sentry: https://sentry.ceph.com/organizations/ceph/?query=753e14e57df34e0d86a81dd4145ce3bf

    Command failed (workunit test kernel_untar_build.sh) on smithi188 with
    status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

[7249336]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:15:05
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249336/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249336/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0bb2c3e3f1124b1b96978254dd378cd7

    Command failed on smithi099 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7249338]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   07:25:05
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249338/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249338/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5e3830c44b314c02ba88e150cba81b02

    Command failed (workunit test suites/fsstress.sh) on smithi142 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7249340]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   03:19:18
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249340/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249340/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fb63b40f0c8d49f4a8fc88b327256a07

    Command failed (workunit test suites/dbench.sh) on smithi169 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh'

[7249341]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:16:31
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249341/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249341/
sentry: https://sentry.ceph.com/organizations/ceph/?query=823652d1fc7b416cb871dc6d95d46b5b

    more than one file system available

[7249343]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   01:13:56
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249343/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249343/
sentry: https://sentry.ceph.com/organizations/ceph/?query=63d0a8c88be34d87b7009c877cb6e10b

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7249345]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249345/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249345/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7ba8db46d944a95a35ef75d13abbc1c

    more than one file system available

[7249346]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
-----------------------------------------------------------------
time:   01:46:08
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249346/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249346/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1a80950a41e4f598457f3b0187f38eb

    Command failed (workunit test suites/dbench.sh) on smithi093 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7249347]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:14:52
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249347/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249347/
sentry: https://sentry.ceph.com/organizations/ceph/?query=833a88e3af1942718d6eac4b5b7a2dbe

    Command failed (workunit test libcephfs/test.sh) on smithi023 with status
    141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

[7249348]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:40:27
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249348/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249348/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f3909c665aa64de2becf8198b0b03f4c

    Test failure: test_repeated_boot (tasks.cephfs.test_failover.TestFailover)

[7249350]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   01:11:58
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249350/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249350/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e799b0df9db049e78add421040759809

    more than one file system available

[7249351]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:59:16
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249351/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249351/
sentry: https://sentry.ceph.com/organizations/ceph/?query=039d9d2cb3a94ac6b0bc0c95d83f3914

    saw valgrind issues

[7249352]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:17:59
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249352/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249352/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab911959c4da458eb72a4c76a9ed85fd

    more than one file system available

[7249355]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   03:22:52
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249355/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249355/
sentry: https://sentry.ceph.com/organizations/ceph/?query=676e47d0883b4322ba500cd52b697e23

    Test failure: test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration),
    test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration)

[7249356]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
-----------------------------------------------------------------
time:   04:33:23
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249356/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249356/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a9cf5e5835c4801984edd9ee7c3e628

    Command failed (workunit test suites/iozone.sh) on smithi177 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh'

[7249358]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:20:50
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249358/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249358/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd040b791e2f48538fb23fbfe3128a9e

    more than one file system available

[7249360]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:15:20
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249360/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249360/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c3831cba9a54b9d868d2ec3af215250

    Command failed on smithi186 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7249362]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   03:35:18
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249362/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249362/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d824a5acccb64c25a1e41931528b2eea

    Command failed (workunit test suites/dbench.sh) on smithi067 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7249363]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:16:55
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249363/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249363/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94dd5da68c534c509c5f2613b3f642e4

    Command failed (workunit test fs/quota/quota.sh) on smithi005 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'

[7249364]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   01:14:25
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249364/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249364/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a023c0649f74fa68b54ec498e915da3

    more than one file system available

[7249365]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   03:17:05
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249365/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249365/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72efcdb2545c4c63a3761cbb641a86be

    Command failed (workunit test suites/dbench.sh) on smithi097 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7249366]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   03:25:09
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249366/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249366/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1855eb811a844cc2ba2321d69edfe7d2

    Command failed (workunit test suites/dbench.sh) on smithi171 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh'

[7249367]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
-----------------------------------------------------------------
time:   01:45:07
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249367/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249367/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4525999641984cf5bc73c0b7ab0fd9eb

    Command failed (workunit test suites/blogbench.sh) on smithi053 with status
    3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'

[7249368]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   01:10:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249368/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249368/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8b3c73313d4f445babd38d6b7f06331a

    Command failed (workunit test fs/test_python.sh) on smithi118 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7249369]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   01:10:42
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249369/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249369/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae46b7da288e4f478fe9a294d013d89e

    more than one file system available

[7249371]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:48:39
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249371/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249371/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7249373]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:15:40
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249373/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249373/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0d5f7df8cc84c949cbd96e5c845229b

    more than one file system available

[7249375]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249375/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249375/
sentry: https://sentry.ceph.com/organizations/ceph/?query=286c911a9282418c842fe54a885aa23d

    Command failed on smithi123 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7249376]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:32:44
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249376/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249376/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b7b35883ac324088bd1f3305b3667ba6

    Test failure: test_repeated_boot (tasks.cephfs.test_failover.TestFailover)

[7249377]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:17:07
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249377/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249377/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d5fdbeb5ed74ef8804192ca1a308974

    more than one file system available

[7249380]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:40:30
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249380/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249380/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7249382]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   07:35:19
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249382/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249382/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81dc0b06af934e2091f336febd4b5f2a

    Command failed (workunit test suites/fsstress.sh) on smithi027 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7249383]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   03:17:28
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249383/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249383/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7954a27d5f904d8ea0a9a3a4bde99626

    Command failed (workunit test suites/dbench.sh) on smithi096 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh'

[7249385]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:15:22
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249385/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249385/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dda72c3547fe487a9e5c7245730afd33

    more than one file system available

[7249388]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:35:16
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249388/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249388/

    reached maximum tries (50) after waiting for 300 seconds

[7249390]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   03:24:52
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249390/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249390/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc57de01092b48d9a2dca0082ed43a79

    Test failure: test_mirroring_init_failure_with_recovery
    (tasks.cephfs.test_mirroring.TestMirroring)

[7249391]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:17:04
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249391/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249391/
sentry: https://sentry.ceph.com/organizations/ceph/?query=613c01211ea04b5ab5acb96cbb547ef7

    more than one file system available

[7249394]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:19:59
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249394/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249394/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0a4834df3b944062b78825a62b61dbe1

    more than one file system available

[7249395]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:25:21
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249395/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249395/

    "1682332311.635119 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of
    143157.685375 IOPS exceeded the threshold limit of 500.000000 IOPS for
    osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is
    to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio)
    and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log

[7249396]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:15:09
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249396/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249396/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a4df67ac2834b0bbc6a30e5679537bf

    more than one file system available

[7249398]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:34:50
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249398/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249398/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4e9e4d095c34517a25f8060084cb519

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi078
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0ae1051cf829e759b8b6fc1948ef9ccae79244cf
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'



Dead
=================================================================
[7249332]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249332/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249332/

    hit max job timeout

[7249359]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249359/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249359/

    hit max job timeout

[7249381]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249381/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249381/

    hit max job timeout

[7249387]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249387/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249387/

    hit max job timeout

[7249397]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249397/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249397/

    Error reimaging machines: Failed to power on smithi073



Running
=================================================================
[7249353] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249353/



Pass
=================================================================
[7249328] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   01:21:48
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249328/

[7249330] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
time:   00:46:09
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249330/

[7249337] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:29:39
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249337/

[7249339] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:01:44
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249339/

[7249342] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   02:01:54
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249342/

[7249344] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   01:58:37
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249344/

[7249349] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   02:14:26
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249349/

[7249354] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:38:22
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249354/

[7249357] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}}
time:   00:44:35
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249357/

[7249361] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:26:40
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249361/

[7249370] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:55:37
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249370/

[7249372] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   01:27:19
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249372/

[7249374] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}}
time:   00:29:50
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249374/

[7249378] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:03:12
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249378/

[7249379] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   01:17:22
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249379/

[7249384] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   01:56:47
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249384/

[7249386] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   02:12:37
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249386/

[7249389] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:47:52
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249389/

[7249392] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
time:   00:53:12
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249392/

[7249393] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   01:30:39
info:   http://pulpito.ceph.com/vshankar-2023-04-24_06:32:32-fs-wip-vshankar-testing-20230420.132447-testing-default-smithi/7249393/