2022-12-03T03:35:11.190 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-03T03:35:11.191 INFO:teuthology.results:subset: '63/256'
2022-12-03T03:35:11.192 INFO:teuthology.results:seed: '6582'
2022-12-03T07:55:59.823 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-03T07:55:59.873 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-03T08:47:04.783 INFO:teuthology.results:Tests finished! gathering results...
2022-12-03T08:47:26.600 INFO:scrape:Found 241 jobs
2022-12-03T08:47:38.733 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099754/teuthology.log
2022-12-03T08:48:22.100 INFO:scrape:Found 13 distinct failure reasons
2022-12-03T08:48:22.102 INFO:scrape:Failure: "1670043581.7262516 mon.a (mon.0) 492 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY)" in cluster log
2022-12-03T08:48:22.103 INFO:scrape:3 jobs: ['7099758', '7099887', '7099914']
2022-12-03T08:48:22.104 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2022-12-03T08:48:22.105 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distros/ubuntu_latest', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/metrics}', 'tasks/snap-schedule}']
2022-12-03T08:48:22.105 INFO:scrape:
2022-12-03T08:48:22.106 INFO:scrape:Failure: Command failed on smithi047 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout de0d7fdb293c079dc341d1b2f75089d082eb01fc'
2022-12-03T08:48:22.106 INFO:scrape:157 jobs
2022-12-03T08:48:22.110 INFO:scrape:suites intersection: []
2022-12-03T08:48:22.112 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client', '1-mount', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-upgrade', '2-workload/dbench', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-compat_client', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-flush/yes', '4-verify}', '5-client-sanity}}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'centos_latest', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client-micro', 'clusters/1-node-1-mds-1-osd', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/single-container-host}', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/32bits/{begin/{0-install', 'fs/cephadm/multivolume/{0-start', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'multimds/yes', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'overrides}}', 'pg-warn', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'roles', 'session_timeout', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{rhel_8}', 'syntax}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client}', 'tasks/dbench', 'tasks/fsstress', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/libcephfs/{frag', 'tasks/libcephfs_python}', 'tasks/mgr-osd-full}', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{0-octopus', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/cephfs-mirror-ha-workunit}', 'wsync/no}', 'wsync/yes}']
2022-12-03T08:48:22.113 INFO:scrape:
2022-12-03T08:48:22.114 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-12-03T08:48:22.115 INFO:scrape:1 jobs: ['7099882']
2022-12-03T08:48:22.116 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-12-03T08:48:22.117 INFO:scrape:
2022-12-03T08:48:22.119 INFO:scrape:Dead: None
2022-12-03T08:48:22.119 INFO:scrape:7 jobs: ['7099754', '7099821', '7099712', '7099802', '7099932', '7099918', '7099842']
2022-12-03T08:48:22.120 INFO:scrape:
2022-12-03T08:48:22.120 INFO:scrape:Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)
2022-12-03T08:48:22.121 INFO:scrape:1 jobs: ['7099841']
2022-12-03T08:48:22.121 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/backtrace}']
2022-12-03T08:48:22.122 INFO:scrape:
2022-12-03T08:48:22.122 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)
2022-12-03T08:48:22.123 INFO:scrape:1 jobs: ['7099902']
2022-12-03T08:48:22.123 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-flush}']
2022-12-03T08:48:22.123 INFO:scrape:
2022-12-03T08:48:22.124 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
2022-12-03T08:48:22.124 INFO:scrape:1 jobs: ['7099726']
2022-12-03T08:48:22.125 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}']
2022-12-03T08:48:22.125 INFO:scrape:
2022-12-03T08:48:22.126 INFO:scrape:Failure: Command failed on smithi072 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c5ffcabbeb6e10855afbf933a7627f19fc046fcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ddbf46e-72d2-11ed-843e-001a4aab830c -- ceph orch daemon add osd smithi072:vg_nvme/lv_4'
2022-12-03T08:48:22.126 INFO:scrape:1 jobs: ['7099839']
2022-12-03T08:48:22.127 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2022-12-03T08:48:22.127 INFO:scrape:
2022-12-03T08:48:22.128 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2022-12-03T08:48:22.128 INFO:scrape:1 jobs: ['7099733']
2022-12-03T08:48:22.129 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2022-12-03T08:48:22.129 INFO:scrape:
2022-12-03T08:48:22.130 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
2022-12-03T08:48:22.130 INFO:scrape:1 jobs: ['7099792']
2022-12-03T08:48:22.131 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}']
2022-12-03T08:48:22.131 INFO:scrape:
2022-12-03T08:48:22.132 INFO:scrape:Failure: Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aca831ec-72c6-11ed-843e-001a4aab830c -- ceph mon dump -f json'
2022-12-03T08:48:22.132 INFO:scrape:1 jobs: ['7099757']
2022-12-03T08:48:22.134 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2022-12-03T08:48:22.134 INFO:scrape:
2022-12-03T08:48:22.135 INFO:scrape:Failure: Test failure: test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair)
2022-12-03T08:48:22.135 INFO:scrape:1 jobs: ['7099897']
2022-12-03T08:48:22.136 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/journal-repair}']
2022-12-03T08:48:22.136 INFO:scrape:
2022-12-03T08:48:22.137 INFO:scrape:Failure: {'smithi137.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['python3-libsemanage', 'python3-policycoreutils'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'installed', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'rc': 1, 'results': []}}
2022-12-03T08:48:22.137 INFO:scrape:1 jobs: ['7099710']
2022-12-03T08:48:22.138 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2022-12-03T08:48:22.138 INFO:scrape:
2022-12-03T08:48:22.157 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/
failed:  169
dead:    8
running: 0
waiting: 0
queued:  0
passed:  64


Fail
=================================================================
[7099705]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:19:16
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099705/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099705/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75a987dc38a041e6a23006267886f3d1

    Command failed on smithi040 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099706]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:20:13
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099706/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099706/
sentry: https://sentry.ceph.com/organizations/ceph/?query=03d9399305d045cdaab689f7b222a662

    Command failed on smithi088 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099708]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:40:07
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099708/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099708/
sentry: https://sentry.ceph.com/organizations/ceph/?query=761ae3b9f731497695839fc8f5fd85f3

    Command failed on smithi067 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099709]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:24:33
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099709/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099709/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3df64dac38fb4a4086d2119acc8c4d35

    Command failed on smithi189 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099711]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:25:19
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099711/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099711/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7d9ba72695b2491eb3819017270b805c

    Command failed on smithi044 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099713]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:41:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099713/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099713/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f9bc03269a4a4eabbce311b0846601ca

    Command failed on smithi029 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099714]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:35:07
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099714/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099714/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da53c2dc666b485d98532ebf16cb2727

    Command failed on smithi005 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099715]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:16:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099715/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099715/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b54343af77aa4765bff8bad4d647e534

    Command failed on smithi159 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099716]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:18:29
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099716/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099716/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad7f39179d2b4e7aa07c4638a35e2f31

    Command failed on smithi139 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099718]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:30:19
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099718/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099718/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e597d39dd5e943349c71c672b37d0e38

    Command failed on smithi018 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099719]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:34:09
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099719/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099719/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e43057b863314451ae9075c91492a1b6

    Command failed on smithi092 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099723]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:15:24
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099723/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099723/
sentry: https://sentry.ceph.com/organizations/ceph/?query=15a589475f3d4a669119f7c37314b278

    Command failed on smithi002 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099725]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:15:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099725/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099725/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01cadd8810b84ccea4db9e3068d5e8b2

    Command failed on smithi133 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099726]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   01:16:21
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099726/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099726/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[7099727]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:39
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099727/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099727/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3fcfca3b88374d60aa76a35fb4cff2cd

    Command failed on smithi137 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099729]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:34:35
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099729/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099729/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cafa46eb05349208456fe6e39eb5248

    Command failed on smithi038 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099730]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:27:32
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099730/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099730/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eb04730ce3bd451a80b3c228145f3025

    Command failed on smithi132 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099731]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:30:03
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099731/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099731/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e7ed708fe0344888b7463766e8e7d4ee

    Command failed on smithi049 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099732]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:20:01
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099732/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099732/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e10fcfb4eb4e41ee9a9a38419f3e5190

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099733]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:21:53
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099733/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099733/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ebfe8ea3fac47b6b82a8104aa2b3644

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[7099734]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:16:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099734/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099734/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e6c176cd5ea4e35ba9f17bef11cb07c

    Command failed on smithi140 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099735]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:19:53
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099735/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099735/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbe891dc2d5d43089cebd155856d6ffe

    Command failed on smithi146 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099736]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:23:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099736/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099736/
sentry: https://sentry.ceph.com/organizations/ceph/?query=135d353c6afd4081b67764e8e05847d0

    Command failed on smithi136 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099737]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:34:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099737/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099737/
sentry: https://sentry.ceph.com/organizations/ceph/?query=968e83f8fbc242ed8ba0a53472c2571a

    Command failed on smithi062 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099738]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:15:17
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099738/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099738/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d223a9f6f25c44c6bb82390d489ceaa2

    Command failed on smithi155 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099739]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
-----------------------------------------------------------------
time:   00:19:40
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099739/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099739/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2e8e0e22fd74b42a7df54de72eaede1

    Command failed on smithi171 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099740]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:30:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099740/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099740/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d672df0556e4949b92a3d1d8828fed1

    Command failed on smithi159 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099742]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:34:22
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099742/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099742/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb70ca233ae74ed8b4ef43dff4c1f648

    Command failed on smithi040 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099743]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:20:27
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099743/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099743/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d799d78f31c84e8ab26064ba2fee796e

    Command failed on smithi130 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099744]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:25:26
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099744/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099744/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1dff78f55f7e4ea68ddfc7d60d13eeca

    Command failed on smithi090 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099745]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:30:09
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099745/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=92b5e58b8d664ea6b64dc6dd63bb7e1b

    Command failed on smithi033 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099747]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:34:50
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099747/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07f80066b78546369d3e1cd531267e2c

    Command failed on smithi007 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099748]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099748/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099748/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e1ad76321814f8581839af93433dd65

    Command failed on smithi002 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099750]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:30:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099750/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099750/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7f13f0e684e34a9f86e445cc17369cc2

    Command failed on smithi005 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099751]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:18:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099751/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099751/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5dcaac2bb7244c3587e8fe170bdc5c88

    Command failed on smithi163 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099752]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:34:59
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099752/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9055b27f0814ac28c5e14c1d7e679ed

    Command failed on smithi018 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099755]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:14:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099755/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099755/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b6d59c9f46f49db87ff005fa418aac8

    Command failed on smithi079 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099756]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:33:50
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099756/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099756/
sentry: https://sentry.ceph.com/organizations/ceph/?query=abc7749e91154c7796e0390270d77214

    Command failed on smithi088 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099757]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:13
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099757/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099757/
sentry: https://sentry.ceph.com/organizations/ceph/?query=439cdff720bb4515925560c29d41b575

    Command failed on smithi112 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    aca831ec-72c6-11ed-843e-001a4aab830c -- ceph mon dump -f json'

[7099758]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:58:25
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099758/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099758/

    "1670043581.7262516 mon.a (mon.0) 492 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7099759]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:18:17
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099759/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099759/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6b2f35115f2042f4b441ea848e5439f1

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099761]  fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
-----------------------------------------------------------------
time:   00:25:41
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099761/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099761/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7dd33f6b29f54430a0680b4709cf6cc2

    Command failed on smithi067 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099762]  fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
-----------------------------------------------------------------
time:   00:20:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099762/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099762/
sentry: https://sentry.ceph.com/organizations/ceph/?query=192e76d189264d03b9a30e4fc5916d2e

    Command failed on smithi120 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099763]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
-----------------------------------------------------------------
time:   00:18:22
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099763/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099763/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb9a4d0afda540efa45e9516733c7e00

    Command failed on smithi161 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099765]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:20:35
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099765/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099765/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a9e685209f74cc9a6c4ba27776d7fee

    Command failed on smithi044 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099768]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:19:33
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099768/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099768/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d34f29a82f441ad842c2f326fa156b1

    Command failed on smithi017 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099770]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:15:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099770/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099770/
sentry: https://sentry.ceph.com/organizations/ceph/?query=557133c097e946f4895b68d9e06eaef0

    Command failed on smithi039 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099774]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:31:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099774/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099774/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56595b078bf04342969b760b7c92d0b9

    Command failed on smithi049 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099775]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:34:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099775/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099775/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c1e4ca90e1b473eb2c016e31d908c6f

    Command failed on smithi130 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099776]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:18:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099776/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099776/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7f2fe602330840ec8aeabfe495582129

    Command failed on smithi140 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099778]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:30:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099778/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099778/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05f62aec38284facaf5f84fc19db8723

    Command failed on smithi062 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099779]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:19:05
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099779/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099779/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a163f33f525b41ffb6f3e5dbd6a79f56

    Command failed on smithi092 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099782]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:18:52
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099782/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099782/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffca7971e55645b7a9ac21ada979a804

    Command failed on smithi165 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099783]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:15:29
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099783/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099783/
sentry: https://sentry.ceph.com/organizations/ceph/?query=143633da1412477db2dc865a3724b1a6

    Command failed on smithi102 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099784]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:30:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099784/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099784/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9f2b7e1c9d0949c7a910fe0da442643e

    Command failed on smithi055 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099787]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:35:03
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099787/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099787/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1c851bec5acd42829d91db593a0da471

    Command failed on smithi007 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099788]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:15:47
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099788/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099788/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54c41200c385409f9d90aa4f46993309

    Command failed on smithi154 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099789]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   00:19:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099789/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8fa17d3b16bd410087fb46595352e222

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099790]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:30:24
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099790/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=65ea6a9cc2ad42b5b495b993cd87e951

    Command failed on smithi035 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099791]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:14:30
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099791/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eef1d66369f7477eb8827df40d700db6

    Command failed on smithi161 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099792]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:17:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099792/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099792/
sentry: https://sentry.ceph.com/organizations/ceph/?query=884ee1ef8fc347a3a08a14ab2b828447

    Test failure: test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration)

[7099793]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:34:37
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099793/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4511e43b7736425da9ade5fb6a12559f

    Command failed on smithi005 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099795]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:30:39
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099795/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eaa848a749ea4a24bf409afaf5439de1

    Command failed on smithi039 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099797]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:15:43
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099797/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=876723f2a61e48fba89fc3b99364c133

    Command failed on smithi018 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099798]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:34:03
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099798/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099798/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f446126a07344f7ea5a9b03207be5e6e

    Command failed on smithi026 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099799]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099799/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=85efc4bf774e4beea4b53e430399dc43

    Command failed on smithi067 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099800]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:19:46
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099800/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b6bf7ecb962f47d3a3c3af31fb4e2279

    Command failed on smithi093 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099801]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
-----------------------------------------------------------------
time:   00:20:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099801/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ff3058af2154e24a75a8e1e5a2d6b35

    Command failed on smithi102 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099803]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:24:34
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099803/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=494d167124de417385338eec1547256b

    Command failed on smithi072 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099804]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:34:11
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099804/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099804/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e444340689dc4442a62d028dc58f91e2

    Command failed on smithi089 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099805]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:19:27
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099805/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099805/
sentry: https://sentry.ceph.com/organizations/ceph/?query=26fcf7abcfe344bbb57e87605f5ccc1a

    Command failed on smithi187 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099806]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:19:36
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099806/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099806/
sentry: https://sentry.ceph.com/organizations/ceph/?query=100fff6974524567be972117cbba7d56

    Command failed on smithi064 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099807]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:31:35
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099807/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6fdafc1e9828479b806d11f504cd916c

    Command failed on smithi090 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099810]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099810/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099810/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0cfc1fe8d06e481c95c0f373590b535a

    Command failed on smithi080 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099812]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:34:42
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099812/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099812/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fea38110ff57471f8384a01c81af1838

    Command failed on smithi049 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099813]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:22:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099813/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099813/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1382d8e121c044dc8a9c4fe8b10778c4

    Command failed on smithi148 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099815]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:29:42
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099815/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099815/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7c5126679e314faeba38b0a7bd3f1063

    Command failed on smithi037 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099816]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:20:32
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099816/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099816/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db5270b17f28448380d26cd4c2b06686

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099817]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:25:09
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099817/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099817/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d60c26890d4847cf9ec785fc9e5eb585

    Command failed on smithi018 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099819]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:39:58
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099819/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099819/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02c5b1ce43954d46a2bea2611b101041

    Command failed on smithi062 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099820]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:58
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099820/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099820/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f67495fd1b77435e9a7c082f9fe8de69

    Command failed on smithi017 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099823]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:25:12
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099823/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099823/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6adb59309ee9471bacd1ff4c6de61c3d

    Command failed on smithi006 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099824]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:31:32
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099824/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c28106aa40344700b79129fc5aec56ed

    Command failed on smithi093 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099827]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:16:30
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099827/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099827/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae64fc74faf64b4a9fc86d72245f07a4

    Command failed on smithi002 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099829]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:19:37
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099829/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e31e677f14f4f9dba2200d77ed4bb4d

    Command failed on smithi035 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099830]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:18:40
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099830/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099830/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a4db902f255e42ca91413146f523d73c

    Command failed on smithi158 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099831]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:24:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099831/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099831/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8df90b1c51c94dad9c5827493621977b

    Command failed on smithi102 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099832]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:13:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099832/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099832/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba6d57d46e454cab83bfdd572f531156

    Command failed on smithi154 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099833]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
-----------------------------------------------------------------
time:   00:15:56
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099833/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=04a4f6868414474a9a68d85b1ec6d6d3

    Command failed on smithi044 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099834]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:18:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099834/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099834/
sentry: https://sentry.ceph.com/organizations/ceph/?query=68bddb80c23c46188bbe22df31663c66

    Command failed on smithi039 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099835]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:40:22
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099835/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099835/
sentry: https://sentry.ceph.com/organizations/ceph/?query=68310e1702534579989c38985deaeeea

    Command failed on smithi008 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099837]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:24:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099837/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099837/
sentry: https://sentry.ceph.com/organizations/ceph/?query=444c5fe9edf34ff0a46fcdb9d3115f49

    Command failed on smithi165 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099838]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:18:26
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099838/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099838/
sentry: https://sentry.ceph.com/organizations/ceph/?query=872393d7d13649a8974ffa924e41c04e

    Command failed on smithi175 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099839]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:20:05
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099839/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099839/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b03b8f8e947545bbbcd01b5d51d5f810

    Command failed on smithi072 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:c5ffcabbeb6e10855afbf933a7627f19fc046fcb shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7ddbf46e-72d2-11ed-843e-001a4aab830c -- ceph orch daemon add osd
    smithi072:vg_nvme/lv_4'

[7099840]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:16:11
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099840/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099840/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a35f5bf8d1fd47f3a133aca214ebc72e

    Command failed on smithi170 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099841]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:15:39
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099841/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099841/
sentry: https://sentry.ceph.com/organizations/ceph/?query=056a1ab92a8f44dd931175fdc6b296d3

    Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)

[7099843]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:39:53
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099843/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ae8e696cb354d26bb74fc8708ffad00

    Command failed on smithi089 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099845]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:23:09
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099845/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d9b5d1bb7afb4eb6a5781c9b19cc9f93

    Command failed on smithi038 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099846]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:20:32
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099846/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e8df3c9d53e46358b34a8fe94ec6600

    Command failed on smithi018 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099849]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:19:50
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099849/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099849/
sentry: https://sentry.ceph.com/organizations/ceph/?query=419124cef9e441e3a82c1f2eea8e8390

    Command failed on smithi090 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099851]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:25:01
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099851/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099851/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0afaf3eef90049b09070ee5f7689c6b1

    Command failed on smithi049 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099853]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:40:10
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099853/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099853/
sentry: https://sentry.ceph.com/organizations/ceph/?query=09c6b13c5d5244f7880648364166a527

    Command failed on smithi037 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099854]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:25:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099854/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099854/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed8c6dc598d94dae9d9d67f705e2f073

    Command failed on smithi006 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099855]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:23:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099855/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099855/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e9d1c9d92e934e1e8359610db0f51119

    Command failed on smithi081 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099856]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:13:54
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099856/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099856/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e57c8f8a05b8410ea5d0fe885c66c7f0

    Command failed on smithi133 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099858]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:40:07
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099858/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099858/
sentry: https://sentry.ceph.com/organizations/ceph/?query=114b19da711848a49406987d75e6a0ec

    Command failed on smithi026 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099859]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:20:29
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099859/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c54eab633c94a81ac75dc4b854314cc

    Command failed on smithi005 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099860]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:19:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099860/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099860/
sentry: https://sentry.ceph.com/organizations/ceph/?query=082bd8033dee409eb912c0fd28146607

    Command failed on smithi140 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099861]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:24:31
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099861/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099861/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c417b32632bc47c1a79b36090436918a

    Command failed on smithi102 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099863]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:39:46
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099863/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099863/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3875a2a176f14489b1f6cd042eb68fdc

    Command failed on smithi033 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099864]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099864/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099864/
sentry: https://sentry.ceph.com/organizations/ceph/?query=badee5e10cf74357a02a869f765a2dbb

    Command failed on smithi064 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099865]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:24:27
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099865/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099865/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19f9fc0ac1e246cf843f5c2b5f08ce2c

    Command failed on smithi138 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099866]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:24:50
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099866/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099866/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94cb75b41bdd4ecca8c3b3f12603bda5

    Command failed on smithi161 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099868]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:40:24
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099868/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099868/
sentry: https://sentry.ceph.com/organizations/ceph/?query=051cd6e63c7949e7a25dde65564207f7

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099869]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:16:25
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099869/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099869/
sentry: https://sentry.ceph.com/organizations/ceph/?query=093e772e48b145499fdef4dcf4386cb3

    Command failed on smithi169 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099870]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:19:45
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099870/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099870/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e8fa97db6004cce9335b348eb7bd14d

    Command failed on smithi012 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099872]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:19:05
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099872/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099872/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9f3dbd5bbdb4456d925a1957694e60df

    Command failed on smithi103 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099874]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:13:13
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099874/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099874/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4f827f96a54446e28c592ba00453e72e

    Command failed on smithi112 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099875]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
-----------------------------------------------------------------
time:   00:19:29
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099875/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099875/
sentry: https://sentry.ceph.com/organizations/ceph/?query=594cd971562d4883bedf05dec6dfc4f4

    Command failed on smithi133 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099876]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:39:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099876/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099876/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fcebcca48612423dbab8930334163a89

    Command failed on smithi007 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099878]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:24:42
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099878/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099878/
sentry: https://sentry.ceph.com/organizations/ceph/?query=909d86ee66c04e908139be3c12a3f5ce

    Command failed on smithi154 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099879]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:17:56
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099879/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099879/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30a5b8b1a8f0491ea743de8dc06eb13a

    Command failed on smithi080 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099880]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:23:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099880/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099880/
sentry: https://sentry.ceph.com/organizations/ceph/?query=133eb07ee73f404999bd928b611a4605

    Command failed on smithi134 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099881]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:40:07
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099881/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099881/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d802ff3bb7e48b39f3a4b54803b8eaa

    Command failed on smithi049 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099882]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:16:24
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099882/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099882/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de8b1965b2974472ab2d9cec73443d96

    Test failure: test_backtrace_repair
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7099883]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:24:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099883/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099883/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06feaacb14174c568e97e8cc8f204ce5

    Command failed on smithi008 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099884]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099884/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099884/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60b197cab3714b6983016882bc8cefe7

    Command failed on smithi035 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099886]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:19:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099886/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099886/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b49b8181901b40df9aa28712a8be6311

    Command failed on smithi017 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099887]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   01:07:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099887/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099887/

    "1670051496.8211224 mon.a (mon.0) 365 : cluster [WRN] Health check failed:
    Degraded data redundancy: 27/2050 objects degraded (1.317%), 2 pgs degraded
    (PG_DEGRADED)" in cluster log

[7099889]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:17:43
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099889/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099889/
sentry: https://sentry.ceph.com/organizations/ceph/?query=91328545ca75479eb254cbafdfffacc4

    Command failed on smithi006 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099890]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:16:12
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099890/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099890/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0965cfd9373646caad24d6dc79d2c0f2

    Command failed on smithi169 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099892]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:39:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099892/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=213ae8fcdb9946be83c06d48ca3c8e9c

    Command failed on smithi161 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099893]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:17
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099893/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099893/
sentry: https://sentry.ceph.com/organizations/ceph/?query=69149490c7094e5eb56e992115e2ecc0

    Command failed on smithi012 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099895]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:18:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099895/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099895/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d4059e3d95944d0939b1d914099695f

    Command failed on smithi103 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099896]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:24:09
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099896/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7aab1388781c445d83488907e4bff2e6

    Command failed on smithi071 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099897]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:13:47
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099897/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099897/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ff3b46b99394c0c9db9a3c6d07e0ccc

    Test failure: test_journal_smoke
    (tasks.cephfs.test_journal_repair.TestJournalRepair)

[7099898]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:39:50
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099898/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099898/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0959ec8e253649a9b112b5e36eac231c

    Command failed on smithi037 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099899]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:14:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099899/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099899/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2f81371cd2841e0bad5fb7f5b7a24d9

    Command failed on smithi039 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099900]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:24:10
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099900/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099900/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dabfcfd3102f47449bde4f7b6ebb49db

    Command failed on smithi093 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099902]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:15:44
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099902/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=63e7b918490841b988833e727feded8e

    Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

[7099903]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:40:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099903/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099903/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc851747ad8745be9f2fb2018cd7dab5

    Command failed on smithi029 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099904]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:24:41
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099904/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099904/
sentry: https://sentry.ceph.com/organizations/ceph/?query=50d2879c7f024f4e9cc989623404685e

    Command failed on smithi026 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099905]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099905/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099905/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24e0f937643848eda60cf1e2f4692dff

    Command failed on smithi079 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099907]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:39:43
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099907/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3173fa701244494b801555a6c265be8c

    Command failed on smithi154 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099909]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}}
-----------------------------------------------------------------
time:   00:34:40
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099909/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099909/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a854e555ba9b424da695f2db4ba5fe1e

    Command failed on smithi008 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099911]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:20:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099911/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099911/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c101a7c7d1424c6ca35859e3513050f8

    Command failed on smithi099 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099912]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:29:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099912/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099912/
sentry: https://sentry.ceph.com/organizations/ceph/?query=195fdbc224364310a300ba491f5888f9

    Command failed on smithi012 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099913]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:32:41
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099913/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81763705001446769784be2ecec2d165

    Command failed on smithi083 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099914]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:41:08
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099914/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099914/

    "1670054431.5915563 mon.a (mon.0) 2274 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)"
    in cluster log

[7099915]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:18:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099915/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099915/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30cab22a76d64c2ca6be3ba079441b14

    Command failed on smithi138 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099916]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:20:12
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099916/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fde6b6fb85e34f98bb93ab1588d0d504

    Command failed on smithi006 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099917]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:24:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099917/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cb50e1aa54a4fb990757cc6c386eb1a

    Command failed on smithi203 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099919]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:13:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099919/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9369fcfb4d3e43698abc761905e48144

    Command failed on smithi040 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.1 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd
    /home/ubuntu/cephtest/clone.client.1 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099920]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
-----------------------------------------------------------------
time:   00:15:59
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099920/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bf4fca946b9e4f04a144235a30411ce3

    Command failed on smithi103 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099922]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:16:22
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099922/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc6eb41b64f64bacacaeb9f2da3b112d

    Command failed on smithi007 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099924]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:34:12
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099924/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5c5138ada484f109398c2f7d59b2e25

    Command failed on smithi071 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099925]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:22:41
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099925/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb0be4cc00fc46df9f6be93f83c77a5c

    Command failed on smithi060 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099928]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:13:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099928/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=250d2c479521482d8d8eac3e03f42f4c

    Command failed on smithi090 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099930]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:29:41
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099930/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099930/
sentry: https://sentry.ceph.com/organizations/ceph/?query=26d56de95f72493ebb436e88f3db29d9

    Command failed on smithi072 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099933]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:15:26
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099933/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099933/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b7ce850f8694e3ab9e4a84e6f78082b

    Command failed on smithi093 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099934]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
-----------------------------------------------------------------
time:   00:29:59
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099934/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099934/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8a71ca464a24b98b4513a63af86d693

    Command failed on smithi039 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099936]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:23:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099936/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8472ec9a7ea741d4b48abafc0dddbfc3

    Command failed on smithi035 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099937]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:34:47
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099937/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099937/
sentry: https://sentry.ceph.com/organizations/ceph/?query=922b32fe844240df88a6ab6781dc6529

    Command failed on smithi132 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099938]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:19:01
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099938/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099938/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd0bb8e9cff64668b212489746bc05b9

    Command failed on smithi007 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099940]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
-----------------------------------------------------------------
time:   00:29:31
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099940/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099940/
sentry: https://sentry.ceph.com/organizations/ceph/?query=deb50adb89cb4b248ca89b803808059c

    Command failed on smithi037 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099941]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:34:23
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099941/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099941/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93cfedb5731a45109ae6278cb14ae433

    Command failed on smithi099 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099942]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:18:42
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099942/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099942/
sentry: https://sentry.ceph.com/organizations/ceph/?query=544897a3964544a4adf8f12b30e483d7

    Command failed on smithi047 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099944]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
-----------------------------------------------------------------
time:   00:30:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099944/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099944/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f5685f4a3e174f7bacf38db964b0e64c

    Command failed on smithi090 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'

[7099945]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:17:43
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099945/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f234c6bd630f4db2bce1cd0d0e0fbe37

    Command failed on smithi187 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    de0d7fdb293c079dc341d1b2f75089d082eb01fc'



Dead
=================================================================
[7099710]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:07:54
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099710/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099710/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2dfb65056635450bb5f5dff2d13265b2

    {'smithi137.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['python3-libsemanage',
    'python3-policycoreutils'], 'releasever': None, 'security': False,
    'skip_broken': False, 'state': 'installed', 'update_cache': False,
    'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download
    metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download
    repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried",
    'rc': 1, 'results': []}}

[7099712]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099712/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099712/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099754]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099754/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099754/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099802]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099802/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099802/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099821]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099821/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099821/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099842]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099842/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099842/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099918]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099918/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099918/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds

[7099932]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099932/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099932/

    Error reimaging machines: reached maximum tries (100) after waiting for 600
    seconds



Pass
=================================================================
[7099707] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:24:58
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099707/

[7099717] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:38:31
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099717/

[7099720] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:20:11
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099720/

[7099721] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:19:46
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099721/

[7099722] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:56:32
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099722/

[7099724] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:13:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099724/

[7099728] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:17:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099728/

[7099741] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:15:17
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099741/

[7099746] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:15:53
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099746/

[7099749] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:32:44
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099749/

[7099753] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:16:22
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099753/

[7099760] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:19:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099760/

[7099764] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
time:   02:20:44
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099764/

[7099766] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:07:34
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099766/

[7099767] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:46:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099767/

[7099769] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:56:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099769/

[7099771] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   00:18:36
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099771/

[7099772] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:29:16
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099772/

[7099773] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:26:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099773/

[7099777] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:19:20
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099777/

[7099780] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:26:05
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099780/

[7099781] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}}
time:   01:19:49
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099781/

[7099785] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:41:34
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099785/

[7099786] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:18:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099786/

[7099794] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   00:19:12
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099794/

[7099796] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:14:10
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099796/

[7099808] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:16:39
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099808/

[7099809] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099809/

[7099811] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:33:43
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099811/

[7099814] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:14:54
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099814/

[7099818] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:17:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099818/

[7099822] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:17:08
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099822/

[7099825] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
time:   00:41:14
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099825/

[7099826] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:23:34
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099826/

[7099828] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:18:48
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099828/

[7099836] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:17:26
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099836/

[7099844] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:14:15
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099844/

[7099847] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:21:04
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099847/

[7099848] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:36:49
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099848/

[7099850] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:16:08
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099850/

[7099852] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:27:26
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099852/

[7099857] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:16:53
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099857/

[7099862] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:34:29
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099862/

[7099867] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:29:52
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099867/

[7099871] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:40:30
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099871/

[7099873] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}}
time:   01:06:28
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099873/

[7099877] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:48:30
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099877/

[7099885] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:25:18
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099885/

[7099888] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:59:55
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099888/

[7099891] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:15:37
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099891/

[7099894] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:24:07
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099894/

[7099901] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:18:03
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099901/

[7099906] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:32:49
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099906/

[7099908] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   00:19:06
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099908/

[7099910] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:13:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099910/

[7099921] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}}
time:   01:13:55
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099921/

[7099923] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:27:57
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099923/

[7099926] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:16:46
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099926/

[7099927] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:25:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099927/

[7099929] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:21:13
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099929/

[7099931] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:14:38
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099931/

[7099935] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:16:51
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099935/

[7099939] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:20:39
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099939/

[7099943] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:13:45
info:   http://pulpito.ceph.com/vshankar-2022-12-01_02:33:43-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7099943/