2022-05-25T00:53:37.143 INFO:root:teuthology version: 1.1.1.dev341+g1b30281f
2022-05-25T00:53:37.145 INFO:teuthology.results:subset: '4/32'
2022-05-25T00:53:37.145 INFO:teuthology.results:seed: '6576'
2022-05-25T02:15:14.302 INFO:root:teuthology version: 1.1.1.dev341+g1b30281f
2022-05-25T02:15:14.335 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-05-25T11:29:00.559 INFO:teuthology.results:Tests finished! gathering results...
2022-05-25T11:29:11.303 INFO:scrape:Found 130 jobs
2022-05-25T11:29:55.415 INFO:scrape:Found 18 distinct failure reasons
2022-05-25T11:29:55.416 INFO:scrape:Failure: "2022-05-25T01:29:22.507001+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log
2022-05-25T11:29:55.417 INFO:scrape:78 jobs
2022-05-25T11:29:55.418 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}']
2022-05-25T11:29:55.419 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{osd-asserts', 'overrides/{whitelist_health', 'overrides/{whitelist_health}', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'supported-random-distros$/{ubuntu_latest}', 'tasks/acls}', 'tasks/alternate-pool}', 'tasks/asok_dump_tree}', 'tasks/cephfs-shell}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-readahead}', 'tasks/client-recovery}', 'tasks/dbench', 'tasks/exports}', 'tasks/forward-scrub}', 'tasks/fsstress', 'tasks/ior-shared-file}', 'tasks/journal-repair}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/libcephfs_python}', 'tasks/mdtest}', 'tasks/mgr-osd-full}', 'tasks/mirror}', 'tasks/pool-perm}', 'tasks/snap-schedule}', 'tasks/snapshots}', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/clone}}', 'test/misc}}', 'thrashosds-health', 'traceless/50pc}', 'validater/lockdep}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.420 INFO:scrape:
2022-05-25T11:29:55.420 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2022-05-25T11:29:55.421 INFO:scrape:1 jobs: ['6847389']
2022-05-25T11:29:55.421 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.422 INFO:scrape:
2022-05-25T11:29:55.422 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
2022-05-25T11:29:55.423 INFO:scrape:2 jobs: ['6847502', '6847404']
2022-05-25T11:29:55.423 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{no_client_pidfile', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.424 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.424 INFO:scrape:
2022-05-25T11:29:55.424 INFO:scrape:Failure: "2022-05-25T02:11:15.394527+0000 mds.f (mds.2) 24 : cluster [WRN] client.4711 isn't responding to mclientcaps(revoke), ino 0x3000000305d pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.004525 seconds ago" in cluster log
2022-05-25T11:29:55.425 INFO:scrape:2 jobs: ['6847430', '6847516']
2022-05-25T11:29:55.425 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.426 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.426 INFO:scrape:
2022-05-25T11:29:55.427 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)
2022-05-25T11:29:55.427 INFO:scrape:2 jobs: ['6847420', '6847517']
2022-05-25T11:29:55.428 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'tasks/mds-flush}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.428 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/mds-flush}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.429 INFO:scrape:
2022-05-25T11:29:55.429 INFO:scrape:Failure: Test failure: test_moncap_with_blanket_allow (tasks.cephfs.test_multifs_auth.TestMONCaps)
2022-05-25T11:29:55.430 INFO:scrape:3 jobs: ['6847505', '6847418', '6847465']
2022-05-25T11:29:55.430 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{mon-debug', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.431 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/multifs/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{mon-debug', 'rhel_8}', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.432 INFO:scrape:
2022-05-25T11:29:55.433 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
2022-05-25T11:29:55.434 INFO:scrape:2 jobs: ['6847401', '6847487']
2022-05-25T11:29:55.435 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-05-25T11:29:55.436 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{crc}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-05-25T11:29:55.437 INFO:scrape:
2022-05-25T11:29:55.437 INFO:scrape:Failure: Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks)
2022-05-25T11:29:55.438 INFO:scrape:1 jobs: ['6847446']
2022-05-25T11:29:55.439 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.440 INFO:scrape:
2022-05-25T11:29:55.441 INFO:scrape:Failure: Test failure: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2)
2022-05-25T11:29:55.442 INFO:scrape:1 jobs: ['6847429']
2022-05-25T11:29:55.443 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/multimds_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.444 INFO:scrape:
2022-05-25T11:29:55.445 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)
2022-05-25T11:29:55.445 INFO:scrape:1 jobs: ['6847501']
2022-05-25T11:29:55.446 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.446 INFO:scrape:
2022-05-25T11:29:55.447 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2022-05-25T11:29:55.447 INFO:scrape:2 jobs: ['6847497', '6847415']
2022-05-25T11:29:55.448 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.448 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.449 INFO:scrape:
2022-05-25T11:29:55.449 INFO:scrape:Failure: Command failed on smithi141 with status 1: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm'
2022-05-25T11:29:55.450 INFO:scrape:1 jobs: ['6847402']
2022-05-25T11:29:55.450 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.451 INFO:scrape:
2022-05-25T11:29:55.451 INFO:scrape:Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays)
2022-05-25T11:29:55.451 INFO:scrape:1 jobs: ['6847457']
2022-05-25T11:29:55.452 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.452 INFO:scrape:
2022-05-25T11:29:55.453 INFO:scrape:Dead: b'2022-05-25T08:21:54.511 DEBUG:teuthology.exit:Finished running handlers'
2022-05-25T11:29:55.453 INFO:scrape:6 jobs: ['6847484', '6847411', '6847518', '6847443', '6847399', '6847513']
2022-05-25T11:29:55.454 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.454 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-client', '2-logrotate}', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'begin/{0-install', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'overrides/{no_client_pidfile', 'pg-warn', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/client-recovery}', 'tasks/{0-check-counter', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}']
2022-05-25T11:29:55.455 INFO:scrape:
2022-05-25T11:29:55.455 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-05-25T11:29:55.456 INFO:scrape:1 jobs: ['6847509']
2022-05-25T11:29:55.456 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/forward-scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.457 INFO:scrape:
2022-05-25T11:29:55.457 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi070 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'
2022-05-25T11:29:55.458 INFO:scrape:1 jobs: ['6847463']
2022-05-25T11:29:55.458 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/libcephfs/{frag', 'test}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.459 INFO:scrape:
2022-05-25T11:29:55.460 INFO:scrape:Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)
2022-05-25T11:29:55.461 INFO:scrape:1 jobs: ['6847488']
2022-05-25T11:29:55.462 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/backtrace}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.462 INFO:scrape:
2022-05-25T11:29:55.463 INFO:scrape:Failure: 
2022-05-25T11:29:55.464 INFO:scrape:1 jobs: ['6847448']
2022-05-25T11:29:55.465 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-25T11:29:55.466 INFO:scrape:
2022-05-25T11:29:55.478 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/
failed:  101
dead:    6
running: 0
waiting: 0
queued:  0
passed:  23


Fail
=================================================================
[6847389]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:13:03
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847389/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847389/
sentry: https://sentry.ceph.com/organizations/ceph/?query=df4ac14b676e46dab1edf89881bd4012

    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)

[6847390]  fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full}
-----------------------------------------------------------------
time:   00:22:02
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847390/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847390/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8887bdee44ba4c61b408362b84ab0dee

    "2022-05-25T01:13:25.059423+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847391]  fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
-----------------------------------------------------------------
time:   00:18:55
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847391/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847391/
sentry: https://sentry.ceph.com/organizations/ceph/?query=85181f1ed5ef470883034bd4adfec7a1

    "2022-05-25T01:13:50.295831+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6847392]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:21:38
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847392/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847392/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca05199790244afcbbbf16f6603c522d

    "2022-05-25T01:24:42.916580+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847393]  fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:23:34
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847393/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847393/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86f6c9bcc60a4855845f694e5ca5e274

    "2022-05-25T01:29:45.823795+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847394]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:24:01
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847394/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847394/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b452faf573464a8f8e2dc85c360bb57b

    "2022-05-25T01:29:31.189250+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847397]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:24:55
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847397/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847397/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9be8fe4416884ada8e4f2b1b78218fde

    "2022-05-25T01:29:00.157898+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847401]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   02:28:28
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847401/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847401/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[6847402]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847402/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847402/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0e3e505b4a84b58ac5b2dbdec96ff1d

    Command failed on smithi141 with status 1: 'sudo rpm -ivh --oldpackage
    --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm'

[6847403]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:20:44
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847403/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847403/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95bc901f660946cfbfc7e1854efed964

    "2022-05-25T01:21:26.416661+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847404]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
-----------------------------------------------------------------
time:   00:12:03
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847404/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847404/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb3c5e12a0f541b4aae8c7366e7ec987

    Test failure: test_fragmented_injection
    (tasks.cephfs.test_data_scan.TestDataScan)

[6847405]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:23:12
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847405/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847405/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c105382e2ea54ff0a778d8504d955696

    "2022-05-25T01:29:22.507001+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847406]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:23:30
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847406/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847406/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8915b204c6e8471f8fc016e6bc01343f

    "2022-05-25T01:27:54.726197+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847407]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:24:15
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847407/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847407/
sentry: https://sentry.ceph.com/organizations/ceph/?query=62b8118f96aa4db2bb207c746112ff93

    "2022-05-25T01:29:10.876601+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847408]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:24:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847408/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847408/
sentry: https://sentry.ceph.com/organizations/ceph/?query=16c75e78ce874518b4e45261d6363dd9

    "2022-05-25T01:29:51.149351+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847409]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:23:01
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847409/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847409/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca99ffb947e14a1fa86f60caca536e09

    "2022-05-25T01:28:58.258545+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847412]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:21:33
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847412/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847412/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a256b3a172084f50bacbe44e215b6674

    "2022-05-25T01:25:00.195290+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847413]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:23:48
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847413/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847413/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1432a1bc178240819b48b2f6675188fb

    "2022-05-25T01:28:52.793005+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847414]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:21:05
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847414/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847414/
sentry: https://sentry.ceph.com/organizations/ceph/?query=942982ca7a87450493dfea40c648d3b0

    "2022-05-25T01:21:41.244871+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847415]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:40:40
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847415/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847415/
sentry: https://sentry.ceph.com/organizations/ceph/?query=faa599060a5149629ef948f1c90fb54d

    Command failed (workunit test suites/fsstress.sh) on smithi142 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6847417]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:24:15
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847417/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847417/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c86572b9662046f186182d67fa0a626a

    "2022-05-25T01:30:03.464807+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847418]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:21:56
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847418/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847418/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e2a2111baf0b42ee8faef81259be0e85

    Test failure: test_moncap_with_blanket_allow
    (tasks.cephfs.test_multifs_auth.TestMONCaps)

[6847419]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:40:02
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847419/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847419/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4ef0480136a14a078b2d232bfc27e58a

    "2022-05-25T01:37:28.029936+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847420]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:17:25
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847420/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847420/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42f632e01dc240fba7fb6e0a1c9a1b10

    Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

[6847421]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847421/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847421/
sentry: https://sentry.ceph.com/organizations/ceph/?query=116671ae5216497fb8713eea92214917

    "2022-05-25T01:22:26.586521+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847422]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:56
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847422/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847422/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b26d8c75405842e1bd98f87596396ed8

    "2022-05-25T01:17:16.057871+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847423]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:41
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847423/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847423/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c988dee7ec39461ba2f3d356e6479329

    "2022-05-25T01:17:08.424756+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847424]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:22:23
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847424/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847424/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbef510c36b24e5994d5fffd4e5e1652

    "2022-05-25T01:27:45.878936+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847426]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:25:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847426/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b2978277bcb446e1aa10b265f294ff32

    "2022-05-25T01:29:38.452256+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847429]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847429/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847429/
sentry: https://sentry.ceph.com/organizations/ceph/?query=139f677cd4a541d199bb4a5924ccf32f

    Test failure: test_scrub_backtrace
    (tasks.cephfs.test_multimds_misc.TestScrub2)

[6847430]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:11:37
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847430/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847430/

    "2022-05-25T02:11:15.394527+0000 mds.f (mds.2) 24 : cluster [WRN]
    client.4711 isn't responding to mclientcaps(revoke), ino 0x3000000305d
    pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.004525 seconds ago" in
    cluster log

[6847432]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:19:13
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847432/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847432/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3591fd79a6e4159ab8f3be96904c315

    "2022-05-25T01:21:00.391811+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847433]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:23:44
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847433/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8993fe20a9a4420b2af9f157c772b72

    "2022-05-25T01:29:35.573304+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847434]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:23:54
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847434/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99916e8df11a408a9caf8dd552de329e

    "2022-05-25T01:29:52.581748+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847435]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:23:41
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847435/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847435/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab30d1adbaac4707a970899b7d66cab5

    "2022-05-25T01:29:07.798027+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847438]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:24:25
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847438/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847438/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a1221d3875d41d88c90190f7be8f586

    "2022-05-25T01:30:06.185678+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847439]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:20:28
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847439/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847439/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2fdedf148cc04643bf4705064d72ee5b

    "2022-05-25T01:25:56.653450+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847440]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:23:43
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847440/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847440/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9c5d982618649cebc57a44e040da933

    "2022-05-25T01:28:48.220637+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847441]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:22:19
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847441/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847441/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cd791f3881346cbb1e0cb38630eafb9

    "2022-05-25T01:29:16.734478+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847442]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:19:15
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847442/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847442/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2613e7bccbf9494a9c664e72ed52ff21

    "2022-05-25T01:32:36.607480+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847444]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:19:09
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847444/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847444/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fbbace2f3769485292b5b65750e26d8e

    "2022-05-25T01:40:23.250776+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847445]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:18:03
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847445/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847445/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f3a97e3f11b9484580877811fde90e35

    "2022-05-25T01:44:08.039515+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847446]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
-----------------------------------------------------------------
time:   00:18:44
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847446/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847446/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a4ffbbd9808a410fa43fed84fb036bcc

    Test failure: test_scrub_checks
    (tasks.cephfs.test_scrub_checks.TestScrubChecks)

[6847447]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:19:02
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847447/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847447/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc762d5f987348af8d203fa4404fbc1d

    "2022-05-25T01:45:51.028001+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847448]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:12:42
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847448/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847448/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d98770464304aea847584bbd530bf7d

[6847449]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:19:03
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847449/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847449/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8410a9d8e6f14c0ca7b7658c1cd4be1a

    "2022-05-25T01:46:16.456685+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847452]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:19:18
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847452/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847452/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e6ffd33ec5f466ca8b888abd8a6545f

    "2022-05-25T01:50:39.016601+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847453]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:22:10
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847453/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=68db2afec62b41dabd4ab7a529cc2ec2

    "2022-05-25T01:51:14.516892+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847454]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
-----------------------------------------------------------------
time:   00:19:42
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847454/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847454/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b42ac064d85b47acbd2b5c153dad61a4

    "2022-05-25T01:51:54.771172+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847455]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:21:56
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847455/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847455/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20dc201414db4b60b27711922e563777

    "2022-05-25T01:51:54.418635+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847456]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:19:06
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847456/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847456/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b4d5d88690a4366ac61cbd962455651

    "2022-05-25T01:52:20.234288+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847457]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
-----------------------------------------------------------------
time:   00:17:47
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847457/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847457/
sentry: https://sentry.ceph.com/organizations/ceph/?query=490c3d5ea0634c28a04e3c1b2a80e6b5

    Test failure: test_hardlink_reintegration
    (tasks.cephfs.test_strays.TestStrays)

[6847458]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:34
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847458/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847458/
sentry: https://sentry.ceph.com/organizations/ceph/?query=61bf085dfd834b0cb6e11d904e46f369

    "2022-05-25T01:53:56.776160+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847459]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:44
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847459/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4839631f0f0e4c2bb8238e8a35293c16

    "2022-05-25T01:54:12.991922+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847460]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:19:36
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847460/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847460/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96cf1f627b7648959f28b0eb380e9de9

    "2022-05-25T01:54:01.520683+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847462]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847462/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847462/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0466c126db6544e1981c429a3416c865

    "2022-05-25T01:56:32.302219+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847463]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:17:59
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847463/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847463/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05f97eff7c964dadaa1b49f0f7417992

    Command failed (workunit test libcephfs/test.sh) on smithi070 with status
    141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

[6847464]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:22:19
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847464/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24c7d20d86014d0dbc0739438ebc76c6

    "2022-05-25T02:03:23.709156+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847465]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:36:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847465/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847465/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d9c61e78a134ef1bd5604cce1171d68

    Test failure: test_moncap_with_blanket_allow
    (tasks.cephfs.test_multifs_auth.TestMONCaps)

[6847466]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:19:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847466/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847466/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b942fdd4f509461b9c13b4c7cdff5243

    "2022-05-25T01:59:14.576216+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847468]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:19:27
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847468/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847468/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f74e7b6dd1fb4f9ab84421f4aefc3925

    "2022-05-25T02:01:12.373823+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847469]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:20:15
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847469/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847469/
sentry: https://sentry.ceph.com/organizations/ceph/?query=76b4cb04a1e04b19adba7b82345456ef

    "2022-05-25T02:01:25.163444+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847470]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:20:35
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847470/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847470/
sentry: https://sentry.ceph.com/organizations/ceph/?query=84b3c6fc017241e38a323c80055d6b10

    "2022-05-25T02:03:16.631361+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847471]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:19:58
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847471/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847471/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77ca864ea72a4f2795ba3798b4bf079f

    "2022-05-25T02:00:58.101898+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847473]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
-----------------------------------------------------------------
time:   00:21:02
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847473/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847473/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7bbacfcc748d4973bd1b3c86c29bc780

    "2022-05-25T02:03:04.804046+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847474]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:50
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847474/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847474/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8cd3b38ffcd1423a938936eba61bdb32

    "2022-05-25T02:00:28.518338+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847475]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:19:31
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847475/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847475/
sentry: https://sentry.ceph.com/organizations/ceph/?query=054a3e0669484aa98f4e1e48dfeecec2

    "2022-05-25T02:01:47.935472+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847476]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:20:55
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847476/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847476/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c442aaacf6354670be19b914f79eb189

    "2022-05-25T02:03:15.821058+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847477]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:19:05
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847477/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847477/
sentry: https://sentry.ceph.com/organizations/ceph/?query=47fe463511254298b8cfab4db4c8dbbe

    "2022-05-25T02:03:14.185283+0000 mgr.y (mgr.4118) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847478]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
-----------------------------------------------------------------
time:   00:19:40
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847478/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847478/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f345d5e6ce94b3fad276db8cad7cea5

    "2022-05-25T02:02:01.162236+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847479]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:20:58
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847479/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847479/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d495c491159b4f0fad3a31feb103cf7d

    "2022-05-25T02:03:28.068557+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847480]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:20:38
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847480/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847480/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3f65990332fc48abbac26e6d91b4a429

    "2022-05-25T02:03:09.026699+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847481]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:20:34
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847481/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847481/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13ec5af48b754ef2a32d1f708f01a900

    "2022-05-25T02:03:23.401689+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847483]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:19:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847483/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847483/
sentry: https://sentry.ceph.com/organizations/ceph/?query=55045a441b2643168ce08c760f9eb1dc

    "2022-05-25T02:05:25.042407+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847485]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847485/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847485/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e3ba93061ef3439bb027d2f7abed3eae

    "2022-05-25T02:08:15.467253+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847486]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:19:12
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847486/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847486/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99cc221a46634bf9bbc085137f335f15

    "2022-05-25T02:12:49.924361+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847487]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   03:05:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847487/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847487/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[6847488]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
-----------------------------------------------------------------
time:   00:16:26
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847488/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847488/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d5a657eaa56470abfdb48f5fbbc99fe

    Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)

[6847489]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:17:58
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847489/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847489/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0db4643ce3324fdda78045aa3ab87c1a

    "2022-05-25T02:15:22.594233+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847491]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:19:13
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847491/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847491/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a925296e6abe48d6a55561e7f2527713

    "2022-05-25T02:18:44.683816+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847492]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
-----------------------------------------------------------------
time:   00:18:33
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847492/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847492/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db07d5625b1248969087efce16f09e85

    "2022-05-25T02:19:00.995442+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847494]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:19:15
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847494/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847494/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a4eb4a0eab094280b01e55cb827e123c

    "2022-05-25T02:22:06.259597+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847495]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:19:21
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847495/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847495/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c341fb5e8ab4882993f0256a61609c7

    "2022-05-25T02:22:33.710096+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847496]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:45
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847496/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847496/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14db412fb0794aba80df76e47501fddf

    "2022-05-25T02:24:03.876298+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847497]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:40:29
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847497/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847497/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4037a28be9e74cbca21a98fa36307207

    Command failed (workunit test suites/fsstress.sh) on smithi167 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6847498]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:23
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847498/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847498/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e8ead7119644581a6758d2a4305a7f1

    "2022-05-25T02:24:11.014635+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847499]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:19:37
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847499/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847499/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a160eecdde4a4b389055ee91e3c36f38

    "2022-05-25T02:24:28.673594+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847501]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
-----------------------------------------------------------------
time:   00:12:38
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847501/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847501/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f5918d6dcaa2484d96a50ff60422f7e7

    Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)

[6847502]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
-----------------------------------------------------------------
time:   00:18:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847502/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847502/
sentry: https://sentry.ceph.com/organizations/ceph/?query=495051d0d24d4165bc51fb40366adec6

    Test failure: test_fragmented_injection
    (tasks.cephfs.test_data_scan.TestDataScan)

[6847503]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:19:07
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847503/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847503/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dbddfa52003479994191d1b2111e324

    "2022-05-25T02:28:36.732213+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847504]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:20:13
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847504/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847504/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77128290c5274ee19fc9fd430b0caa38

    "2022-05-25T02:33:42.804396+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847505]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:27:38
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847505/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847505/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1c807722ab59486c918d7c7453cc24e4

    Test failure: test_moncap_with_blanket_allow
    (tasks.cephfs.test_multifs_auth.TestMONCaps)

[6847506]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:19:19
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847506/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847506/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8448736fb43e422598c7993530932709

    "2022-05-25T02:32:22.872251+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847507]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
-----------------------------------------------------------------
time:   00:18:54
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847507/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847507/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5646f191f26249d791cbb2331758817b

    "2022-05-25T02:32:55.897881+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847509]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:13:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847509/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847509/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1f37a2a0f75f435087bacee265c8342f

    Test failure: test_apply_tag
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[6847510]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:35
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847510/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847510/
sentry: https://sentry.ceph.com/organizations/ceph/?query=674672ce5ad64fe1ab8851739d1f674a

    "2022-05-25T02:31:59.690029+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847511]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:19:48
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847511/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847511/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30c830c2b2ec4ec9adc49d372fd1ca91

    "2022-05-25T02:34:14.473408+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847512]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:19:43
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847512/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847512/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f9de1f4209be4affa199350fbf9241f0

    "2022-05-25T02:33:14.280344+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847514]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:19:23
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847514/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847514/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c10aff0011b24963a5f2e53d81bcc705

    "2022-05-25T02:35:48.036929+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6847516]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:36:30
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847516/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847516/

    "2022-05-25T03:31:35.464969+0000 mds.b (mds.1) 26 : cluster [WRN]
    client.4770 isn't responding to mclientcaps(revoke), ino 0x2000000b1a6
    pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.004054 seconds ago" in
    cluster log

[6847517]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:12:23
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847517/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847517/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dcfe53e8970417f82f921eff85713c8

    Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)



Dead
=================================================================
[6847399]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847399/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847399/

    hit max job timeout

[6847411]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847411/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847411/

    hit max job timeout

[6847443]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847443/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847443/

    hit max job timeout

[6847484]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847484/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847484/

    hit max job timeout

[6847513]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847513/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847513/

    hit max job timeout

[6847518]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847518/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847518/

    hit max job timeout



Pass
=================================================================
[6847395] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:36:17
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847395/

[6847396] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:23:43
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847396/

[6847398] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   01:25:08
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847398/

[6847400] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:22:06
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847400/

[6847410] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:41:14
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847410/

[6847416] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:24:05
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847416/

[6847425] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:23:44
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847425/

[6847427] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
time:   01:10:20
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847427/

[6847428] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   02:51:59
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847428/

[6847431] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:44:01
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847431/

[6847436] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:25:58
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847436/

[6847437] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:43:55
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847437/

[6847450] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:41:07
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847450/

[6847451] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:19:37
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847451/

[6847461] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:28:28
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847461/

[6847467] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:35:42
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847467/

[6847472] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:19:13
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847472/

[6847482] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:32:45
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847482/

[6847490] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:15:52
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847490/

[6847493] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits}
time:   00:19:16
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847493/

[6847500] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:27:39
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847500/

[6847508] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:35:03
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847508/

[6847515] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:42:10
info:   http://pulpito.ceph.com/vshankar-2022-05-24_14:47:38-fs-wip-vshankar-testing-20220523-112412-testing-default-smithi/6847515/