2022-05-27T03:20:15.771 INFO:root:teuthology version: 1.1.1.dev346+g91acb46d
2022-05-27T03:20:15.772 INFO:teuthology.results:seed: '7505'
2022-05-27T13:52:33.159 INFO:root:teuthology version: 1.1.1.dev346+g91acb46d
2022-05-27T13:52:33.179 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-05-27T14:24:36.350 INFO:teuthology.results:Tests finished! gathering results...
2022-05-27T14:24:50.767 INFO:scrape:Found 222 jobs
2022-05-27T14:25:16.410 INFO:scrape:Found 2 distinct failure reasons
2022-05-27T14:25:16.411 INFO:scrape:Failure: "2022-05-27T13:57:52.368913+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log
2022-05-27T14:25:16.411 INFO:scrape:221 jobs
2022-05-27T14:25:16.413 INFO:scrape:suites intersection: ['conf']
2022-05-27T14:25:16.414 INFO:scrape:suites union: ['ceph/ceph', 'clusters/3-node', 'clusters/fixed-1', 'clusters/fixed-3', 'clusters/separate-client', 'conf', 'features/no-object-map', 'features/object-map', 'filestore-xfs', 'install/ceph', 'kernels/single-major-off', 'kernels/single-major-on', 'krbd/basic/{bluestore-bitmap', 'krbd/fsx/{ceph/ceph', 'krbd/ms_modeless/{bluestore-bitmap', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'krbd/singleton/{bluestore-bitmap', 'krbd/thrash/{bluestore-bitmap', 'krbd/unmap/{ceph/ceph', 'krbd/wac/sysfs/{bluestore-bitmap', 'krbd/wac/wac/{bluestore-bitmap', 'ms_mode$/{crc-rxbounce}', 'ms_mode$/{crc}', 'ms_mode$/{legacy-rxbounce}', 'ms_mode$/{legacy}', 'ms_mode$/{secure}', 'ms_mode/crc$/{crc-rxbounce}', 'ms_mode/crc$/{crc}', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/legacy$/{legacy}', 'ms_mode/secure', 'msgr-failures/few', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'randomized-striping-on}', 'striping/default/{msgr-failures/few', 'striping/default/{msgr-failures/many', 'striping/fancy/{msgr-failures/few', 'tasks/fsx-1-client}', 'tasks/fsx-3-client}', 'tasks/krbd_data_pool}', 'tasks/krbd_deep_flatten}', 'tasks/krbd_default_map_options}', 'tasks/krbd_discard}', 'tasks/krbd_exclusive_option}', 'tasks/krbd_fallocate}', 'tasks/krbd_huge_image}', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/krbd_modprobe}', 'tasks/krbd_msgr_segments}', 'tasks/krbd_namespaces}', 'tasks/krbd_parent_overlap}', 'tasks/krbd_read_only}', 'tasks/krbd_rxbounce}', 'tasks/krbd_udev_enumerate}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/krbd_udev_netns}', 'tasks/krbd_udev_symlinks}', 'tasks/krbd_whole_object_zeroout}', 'tasks/rbd_concurrent}', 'tasks/rbd_fio}', 'tasks/rbd_huge_tickets}', 'tasks/rbd_image_read}', 'tasks/rbd_kernel}', 'tasks/rbd_map_snapshot_io}', 'tasks/rbd_map_unmap}', 'tasks/rbd_simple_big}', 'tasks/rbd_workunit_kernel_untar_build}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_ffsb}', 'tasks/rbd_workunit_suites_fsstress_ext4}', 'tasks/rbd_workunit_suites_fsstress}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_suites_iozone}', 'tasks/rbd_workunit_trivial_sync}', 'tasks/rbd_xfstests}', 'tasks/stable_writes}', 'tasks/unmap}', 'tasks/wac', 'thrashers/backoff', 'thrashers/mon-thrasher', 'thrashers/pggrow', 'thrashers/upmap', 'thrashosds-health', 'verify/many-resets}', 'verify/no-resets}', 'workloads/rbd_fio}', 'workloads/rbd_workunit_suites_ffsb}']
2022-05-27T14:25:16.414 INFO:scrape:
2022-05-27T14:25:16.415 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major
2022-05-27T14:25:16.415 INFO:scrape:1 jobs: ['6849740']
2022-05-27T14:25:16.415 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}']
2022-05-27T14:25:16.416 INFO:scrape:
2022-05-27T14:25:16.427 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/
failed:  222
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6849733]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten}
-----------------------------------------------------------------
time:   00:16:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849733/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849733/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9408ffa1bcf144e287e7eb7c36d92904

    "2022-05-27T03:38:59.898880+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849734]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:27:13
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849734/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849734/
sentry: https://sentry.ceph.com/organizations/ceph/?query=32730a114dc84e4f9ab16de877128dd1

    "2022-05-27T04:01:05.604705+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849735]  krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_default_map_options}
-----------------------------------------------------------------
time:   00:26:07
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849735/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849735/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a87f2949473240e4a1a7870fa83025ea

    "2022-05-27T04:00:20.011276+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849736]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:27:25
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849736/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849736/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8d7fdbac27749e5b7a31568659a6978

    "2022-05-27T04:01:12.207625+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849737]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:21:03
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849737/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849737/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7245182eb77943f99dd00629bbaf3f56

    "2022-05-27T03:51:56.425534+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849738]  krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc-rxbounce} msgr-failures/few tasks/rbd_xfstests}
-----------------------------------------------------------------
time:   00:28:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849738/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849738/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4791fb5034f047b3a6c7d9c7640c0d71

    "2022-05-27T04:01:31.504227+0000 mon.a (mon.0) 85 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849739]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/backoff thrashosds-health workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:27:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849739/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849739/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86db8e6d560b4e1098cfba7823513876

    "2022-05-27T04:01:05.776509+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849740]  krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849740/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849740/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e17cc5feec5a403bbc0ac1769c781ade

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re
    f=nightly_pre-single-major

[6849741]  krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes}
-----------------------------------------------------------------
time:   00:16:01
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849741/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849741/
sentry: https://sentry.ceph.com/organizations/ceph/?query=529eeab3d2c84d1fb223add79c8397ed

    "2022-05-27T03:39:40.527138+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849742]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:22:35
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849742/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849742/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1a4a24a0acb48be8db8818106064b06

    "2022-05-27T03:55:31.966769+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849743]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:27:23
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849743/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849743/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2844c4f50ad443baf60318884b12a05

    "2022-05-27T03:55:59.003028+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849744]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:26:03
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849744/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849744/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbdda4ffc7bf4408b0f5f3c6cb065942

    "2022-05-27T03:59:46.257079+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849745]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:26:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849745/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96f1587597404c959d6555c519bae344

    "2022-05-27T04:00:54.871358+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849746]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:21:18
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849746/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849746/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77f3ccd58a184fafb883f96c018a0c47

    "2022-05-27T03:52:59.216686+0000 mgr.y (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849747]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:21:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849747/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=31bdcf9658b8423688a15866925a3983

    "2022-05-27T03:51:50.812457+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849748]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_discard}
-----------------------------------------------------------------
time:   00:16:49
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849748/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849748/
sentry: https://sentry.ceph.com/organizations/ceph/?query=701990f28b7f418791d3f9dc59c9eeea

    "2022-05-27T03:47:31.221714+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849749]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:19:52
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849749/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849749/
sentry: https://sentry.ceph.com/organizations/ceph/?query=466d276dc4064064891ea20794945b5b

    "2022-05-27T03:51:16.236715+0000 mon.a (mon.0) 121 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849750]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:31:12
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849750/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849750/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eabdbd88763341be8d061eb5f374386b

    "2022-05-27T04:00:29.951790+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849751]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:31:56
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849751/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849751/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f455146a7bc741abb0067912d4570440

    "2022-05-27T04:01:00.359244+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849752]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:27:23
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849752/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6138619040d14825a6473f1a45eb7ca0

    "2022-05-27T04:00:42.302895+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849753]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:21:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849753/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849753/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2847386cc19e45e6a2286f636092ff73

    "2022-05-27T03:51:33.010144+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849754]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:27:29
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849754/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849754/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90a28f3653434d00b35ff2a6e4ed4971

    "2022-05-27T04:01:04.145440+0000 mon.a (mon.0) 107 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849755]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:26:42
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849755/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849755/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d91c50cde0014e2a83bd031129418582

    "2022-05-27T04:00:37.022254+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849756]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image}
-----------------------------------------------------------------
time:   00:18:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849756/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849756/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2f4d1f3859a04310959d4abd50706935

    "2022-05-27T03:52:34.423429+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849757]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:23:10
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849757/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849757/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c4ed66669f9846e8b375d80770dc0fd0

    "2022-05-27T03:55:23.839722+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849758]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:27:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849758/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849758/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a51eb81171040e0aa8dc7e6c281c79e

    "2022-05-27T04:01:05.993625+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849759]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:31:23
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849759/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849759/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ccad05380438483b9f6e8613be0233b9

    "2022-05-27T04:00:31.612008+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849760]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:22:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849760/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849760/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f203d41b86024c39a9654d493eeaa1e3

    "2022-05-27T03:55:36.398979+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849761]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:26:43
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849761/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849761/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b25c38732fc44ed85e820b1e0cb2078

    "2022-05-27T04:01:05.098266+0000 mon.a (mon.0) 105 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849762]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:27:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849762/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849762/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4c59cdcfe4c42e5b0793eb3f95e2ec5

    "2022-05-27T04:01:04.243307+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849763]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:19:52
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849763/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849763/
sentry: https://sentry.ceph.com/organizations/ceph/?query=91c02230360340c5a8f7f5254f68f603

    "2022-05-27T03:50:19.170787+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849764]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_modprobe}
-----------------------------------------------------------------
time:   00:16:46
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849764/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849764/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f3d08d5eea054db2a915ff87770bcb4b

    "2022-05-27T03:47:27.318358+0000 mon.a (mon.0) 81 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849765]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:21:12
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849765/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849765/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7bf6a71e1d346cba63ca59569e05158

    "2022-05-27T03:54:50.453988+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849766]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:22:21
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849766/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849766/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c98c30cc2e64e65a9488250cb4ae85b

    "2022-05-27T04:01:14.998169+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849767]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:27:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849767/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849767/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d33d873967a6443a93982c6633f37ec5

    "2022-05-27T04:01:28.454663+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849768]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:26:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849768/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849768/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c0f6c4aa7b6407a9d469b3a68a41e20

    "2022-05-27T04:01:39.082903+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849769]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:20:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849769/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849769/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b140e7d3059447a3a84172984b8f1147

    "2022-05-27T04:01:32.038039+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849770]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:18:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849770/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849770/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db5f8c19402d4623b3c040b42e08fff2

    "2022-05-27T04:01:49.653037+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849771]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:17:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849771/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849771/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45e2f68504ae4e909796fd18d40c4a59

    "2022-05-27T04:03:33.041315+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849772]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:17:31
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849772/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849772/
sentry: https://sentry.ceph.com/organizations/ceph/?query=820e1680330e4bc2b8ef344ea26064c8

    "2022-05-27T04:02:26.386073+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849773]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_msgr_segments}
-----------------------------------------------------------------
time:   00:16:45
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849773/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849773/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7b5afb56e2ce45618832dbde7e846c29

    "2022-05-27T04:01:23.372457+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849774]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:14
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849774/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849774/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58fd9f604d1643c885bc8c463e8e8622

    "2022-05-27T04:09:22.445236+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849775]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:22:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849775/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849775/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e022e4e37ffb4624b013ff86918ff87f

    "2022-05-27T04:13:36.121681+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849776]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:21:57
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849776/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849776/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a2708ca051f4f7797b43569ca4f56da

    "2022-05-27T04:18:16.077204+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849777]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849777/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849777/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e955f36906914e95a6571a0302b8e7f3

    "2022-05-27T04:17:41.531342+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849778]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:17:46
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849778/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849778/
sentry: https://sentry.ceph.com/organizations/ceph/?query=973a26ced01f41398b5c8c4ce4f9b334

    "2022-05-27T04:20:04.247406+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849779]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:22:32
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849779/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849779/
sentry: https://sentry.ceph.com/organizations/ceph/?query=191a42825f954e10ae84aa960f4c8e91

    "2022-05-27T04:19:53.066362+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849780]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:17:56
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849780/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849780/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a44b9cee2a9440b8b5a0dbcd3f38f3d0

    "2022-05-27T04:20:58.622606+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849781]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:17:59
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849781/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849781/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b6953b393fe41d5b69a29205503283e

    "2022-05-27T04:21:08.476484+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849782]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:22:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849782/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849782/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b1c094f3fa647e09b942645dca7676f

    "2022-05-27T04:21:00.176339+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849783]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap}
-----------------------------------------------------------------
time:   00:16:39
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849783/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849783/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ccdf8b8b9f14e3cb270d5becc94501e

    "2022-05-27T04:21:46.737868+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849784]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:16:32
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849784/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849784/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a132af4967d48cdadebc9fb26dab95b

    "2022-05-27T04:23:04.330838+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849785]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:17:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849785/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849785/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a2d47519744d4c5ea59233621ae4d38b

    "2022-05-27T04:23:15.139970+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849786]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:11
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849786/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849786/
sentry: https://sentry.ceph.com/organizations/ceph/?query=161794e97bba45faa44a4df1f4b946c8

    "2022-05-27T04:23:54.220289+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849787]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:23:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849787/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849787/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0868255561e1445badb8d41ad5b90830

    "2022-05-27T04:33:27.331664+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849788]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:17:39
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849788/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849788/
sentry: https://sentry.ceph.com/organizations/ceph/?query=83a37596ce0f49d484da024806ccafee

    "2022-05-27T04:27:35.441177+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849789]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:17:31
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849789/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4453b32400b54bddb8abfef3906b3e61

    "2022-05-27T04:28:46.013414+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849790]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:18:34
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849790/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=27c26d8fd3c841d88ee2c2765910b529

    "2022-05-27T04:33:02.031216+0000 mgr.x (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849791]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_read_only}
-----------------------------------------------------------------
time:   00:15:53
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849791/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8539265252f447a986feb814670111ca

    "2022-05-27T04:29:31.582096+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849792]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849792/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849792/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c6c5b325a01f4558be11bfb248f72d9e

    "2022-05-27T04:34:18.775267+0000 mon.a (mon.0) 115 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849793]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:18:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849793/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e9e4630cefa49f3a7dc7e320c2090af

    "2022-05-27T04:33:01.630700+0000 mgr.x (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849794]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849794/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849794/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ad4e3f9cc9347d7813fd9e5377d6359

    "2022-05-27T04:31:16.561734+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849795]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:18:45
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849795/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b3aa93045c24bbbb34ab97f6dffcd2b

    "2022-05-27T04:35:47.472978+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849796]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:18:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849796/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849796/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3df111016e49484c9e3276aced53394c

    "2022-05-27T04:33:50.982441+0000 mon.a (mon.0) 168 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849797]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:19:03
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849797/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec4dd77e41bd4fbf81f7c1d5aa35f496

    "2022-05-27T04:36:05.938560+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849798]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:19:50
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849798/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849798/
sentry: https://sentry.ceph.com/organizations/ceph/?query=217ea764c6014c4b8a396bfa9fb95a92

    "2022-05-27T04:37:52.701822+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849799]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:24:05
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849799/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=045d0bfc2c3049e39597ca42fb6429b6

    "2022-05-27T04:36:48.158078+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849800]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_whole_object_zeroout}
-----------------------------------------------------------------
time:   00:16:45
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849800/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3496e778500e4b07871063f5ce04316e

    "2022-05-27T04:34:24.581606+0000 mon.a (mon.0) 79 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849801]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:18:47
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849801/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eebe54d656b8421eaa112f7b973cf194

    "2022-05-27T04:35:28.687960+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849802]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:19:40
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849802/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849802/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8459eec570b44941a5e83f50ad0368fc

    "2022-05-27T04:37:22.153930+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849803]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:19:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849803/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72eca0ce7dd54649a2978f877d217da9

    "2022-05-27T04:37:15.619813+0000 mgr.x (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849804]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:20:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849804/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849804/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b392555ee25c40d490231dd77071abb0

    "2022-05-27T04:39:09.994034+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849805]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:21:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849805/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849805/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9df23230029545f1beebd100b62cfd8f

    "2022-05-27T04:38:45.193957+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849806]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:20:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849806/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849806/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8d0fe2c65ec0415a9eb5f56800678fac

    "2022-05-27T04:38:36.456114+0000 mon.a (mon.0) 92 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849807]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:20:50
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849807/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=230b75e6a5c44cf9bc57c158386f4485

    "2022-05-27T04:39:48.663579+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849808]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten}
-----------------------------------------------------------------
time:   00:17:18
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849808/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849808/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a8a5a16e779493a95760572bfefc8da

    "2022-05-27T04:37:28.355244+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849809]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:19:10
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849809/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849809/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dc72e1265b245a59e950cc57934f2ba

    "2022-05-27T04:39:29.165894+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849810]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849810/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849810/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1bfe5c23d41942fab8cf400f7081e50b

    "2022-05-27T04:39:47.899135+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849811]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:20:17
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849811/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849811/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9516650a01484e5f8aa8af26dec8b25d

    "2022-05-27T04:40:07.415521+0000 mon.a (mon.0) 104 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849812]  krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap}
-----------------------------------------------------------------
time:   00:18:18
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849812/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849812/
sentry: https://sentry.ceph.com/organizations/ceph/?query=601ea719b245416a94390450a90ccaa0

    "2022-05-27T04:39:02.085468+0000 mon.a (mon.0) 63 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849813]  krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets}
-----------------------------------------------------------------
time:   00:24:51
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849813/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849813/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a8703b54db849e9b4f0eac1febe9752

    "2022-05-27T04:40:03.207502+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849814]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:20:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849814/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849814/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7810becbf9e947c882d850d0f4e97cc3

    "2022-05-27T04:40:09.602914+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849815]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:17:14
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849815/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849815/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56373267254c4c7992aaebb013101543

    "2022-05-27T04:43:05.947547+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849816]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:16:57
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849816/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849816/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1a958ae3fb2d42ae82990352386fbfc7

    "2022-05-27T04:45:59.982772+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849817]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:17:18
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849817/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849817/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed487f57846f4d56a0556fe26e43ce77

    "2022-05-27T04:45:40.669741+0000 mon.a (mon.0) 129 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849818]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:17:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849818/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849818/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fa01d8c77a1849ccbcc179f445491d3d

    "2022-05-27T04:46:56.659488+0000 mgr.y (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849819]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:17:29
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849819/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849819/
sentry: https://sentry.ceph.com/organizations/ceph/?query=192f8d99d914444a83d685bff206a00a

    "2022-05-27T04:47:54.823072+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849820]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_discard}
-----------------------------------------------------------------
time:   00:16:06
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849820/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849820/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94ab065f5b204a58992026afc6022e11

    "2022-05-27T04:46:43.601896+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849821]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:18:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849821/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849821/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a891a24be7c34c76bdcdaf0c304d349d

    "2022-05-27T04:52:17.987482+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849822]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:13
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849822/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849822/
sentry: https://sentry.ceph.com/organizations/ceph/?query=905b5f2cad25471d8363c668242d0a15

    "2022-05-27T04:50:14.242719+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849823]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:17:47
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849823/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849823/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8bdac978dde64aa79a3236879356f96a

    "2022-05-27T04:52:23.399911+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849824]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:22:34
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849824/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7ec9542132d4dc39e985452954f4b98

    "2022-05-27T04:52:29.085571+0000 mon.a (mon.0) 101 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849825]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:17:18
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849825/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849825/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40a695551ffd4edb98b21027772b159f

    "2022-05-27T04:51:36.717032+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849826]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:17:51
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849826/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849826/
sentry: https://sentry.ceph.com/organizations/ceph/?query=facb8feb11cb410ead1227be03fe6e29

    "2022-05-27T04:52:29.978063+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849827]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:17:27
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849827/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849827/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c17bf86607e64294b09ec31e60c30812

    "2022-05-27T04:53:31.864490+0000 mgr.x (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849828]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:17:46
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849828/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849828/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c5ba9d6ae59f4a0ba7f410049a689a3c

    "2022-05-27T04:53:38.078369+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849829]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_huge_image}
-----------------------------------------------------------------
time:   00:15:53
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849829/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=256edc9a80f448e7991e886d5600b690

    "2022-05-27T04:52:45.528109+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849830]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:16:07
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849830/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849830/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9797fd0a94ee4fcdaeb244447c0de1c0

    "2022-05-27T04:54:15.705699+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849831]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:17:11
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849831/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849831/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e013af3c94224910b81886e7348cf08a

    "2022-05-27T04:56:37.936121+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849832]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849832/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849832/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cec451e0466b42cb8e4215e6781d458f

    "2022-05-27T04:58:56.789873+0000 mgr.x (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849833]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:17:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849833/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80963daec2d34fda887081a025920137

    "2022-05-27T05:01:15.162240+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849834]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:21:48
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849834/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849834/
sentry: https://sentry.ceph.com/organizations/ceph/?query=48b7732b08814544a8ddf554173dd5b9

    "2022-05-27T05:01:30.075765+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849835]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849835/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849835/
sentry: https://sentry.ceph.com/organizations/ceph/?query=98d69a3ac3bd4d148bdc60bad4884757

    "2022-05-27T05:03:08.561335+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849836]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:17:09
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849836/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849836/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f00ab9d2b83d4f47bc04d3829702c71a

    "2022-05-27T05:03:10.593665+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849837]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe}
-----------------------------------------------------------------
time:   00:15:42
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849837/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849837/
sentry: https://sentry.ceph.com/organizations/ceph/?query=944ed40dacf94b68873e9db141f56a9f

    "2022-05-27T05:01:54.985684+0000 mon.a (mon.0) 81 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849838]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:17:49
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849838/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849838/
sentry: https://sentry.ceph.com/organizations/ceph/?query=045e2cf2874e4eea921ff59c82f5d5c7

    "2022-05-27T05:04:27.293180+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849839]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:22:22
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849839/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849839/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20fd4576130b4bd0b06f22363c198adf

    "2022-05-27T05:04:42.973189+0000 mon.a (mon.0) 102 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849840]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849840/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849840/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6d4046b0c3b14ae7ab863505aafe6bf6

    "2022-05-27T05:04:53.429073+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849841]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:18:03
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849841/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849841/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c3a4133e021d49d7b3f0fefeefef0f32

    "2022-05-27T05:06:18.267031+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849842]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:17:45
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849842/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849842/
sentry: https://sentry.ceph.com/organizations/ceph/?query=83e838a7fa9d490b8e48143bde056c8f

    "2022-05-27T05:06:42.941528+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849843]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:23:12
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849843/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2f5d96f59cbc414cbd8e3f3c479d82ed

    "2022-05-27T05:08:24.128723+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849844]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:18:30
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849844/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849844/
sentry: https://sentry.ceph.com/organizations/ceph/?query=49e67518601842a6b3898b158e994b94

    "2022-05-27T05:08:24.668425+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849845]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_msgr_segments}
-----------------------------------------------------------------
time:   00:15:59
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849845/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fdd2817810fc4345a1aeefee169cb822

    "2022-05-27T05:04:36.987606+0000 mon.a (mon.0) 79 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849846]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:19:27
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849846/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40b330ac83594bd892f6db3b17b00aa5

    "2022-05-27T05:11:18.339523+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849847]  krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce}
-----------------------------------------------------------------
time:   00:17:48
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849847/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97e674b8f7954926857b6cd066ec5637

    "2022-05-27T05:07:12.328805+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849848]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:18:25
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849848/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849848/
sentry: https://sentry.ceph.com/organizations/ceph/?query=adc0dff64f0d4c9eba3bb1ff7e50ec29

    "2022-05-27T05:08:26.460870+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849849]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:18:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849849/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849849/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f77c3c7ce87646839a2f3a95c479e5d8

    "2022-05-27T05:09:56.810035+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849850]  krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy-rxbounce} msgr-failures/many tasks/rbd_xfstests}
-----------------------------------------------------------------
time:   00:20:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849850/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849850/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75fe039897af448aaac0a5124d06673a

    "2022-05-27T05:11:18.598069+0000 mon.a (mon.0) 85 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849851]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:18:22
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849851/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849851/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c5bf5123093f44a7b2e6b124615e408a

    "2022-05-27T05:10:07.790785+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849852]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:18:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849852/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849852/
sentry: https://sentry.ceph.com/organizations/ceph/?query=92bbb17a75b7418e806c6b34609eaced

    "2022-05-27T05:11:48.564462+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849853]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:19:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849853/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849853/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb79be3ad4ae4d0b92e25e5069ab1b1e

    "2022-05-27T05:11:06.438334+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849854]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:23:43
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849854/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849854/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4336f6a0990447c9baa9ae5e4e2c009d

    "2022-05-27T05:11:36.646244+0000 mon.a (mon.0) 125 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849855]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849855/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849855/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c32f293ae7fa4683826bbe0777ba3684

    "2022-05-27T05:11:32.303153+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849856]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:23:05
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849856/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849856/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1f15850531bf48d4a0008f1a463012e6

    "2022-05-27T05:11:41.892354+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849857]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:22:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849857/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849857/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2bfcd51329ba401da57b7da07f41be56

    "2022-05-27T05:12:41.838775+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849858]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_parent_overlap}
-----------------------------------------------------------------
time:   00:16:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849858/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849858/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7331fe99221e4556b81873f975df1157

    "2022-05-27T05:11:54.546261+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849859]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:16:22
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849859/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=28f6778774a04375a2a2a4bca957a980

    "2022-05-27T05:12:50.162600+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849860]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:17:09
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849860/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849860/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2295f6778a304e6cba4f1ff73c289fba

    "2022-05-27T05:14:04.623571+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849861]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:29
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849861/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849861/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e108a2e742b4071b75fe22afa2eab3b

    "2022-05-27T05:15:58.132319+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849862]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849862/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849862/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a75a997deb904f6ba25c5fa0ecf0030d

    "2022-05-27T05:15:54.569602+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849863]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849863/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849863/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02739ca88b1c4ad78fcbdaffe82c701c

    "2022-05-27T05:17:30.266663+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849864]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849864/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849864/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c63395be6ae4d0fa314979b69e115f3

    "2022-05-27T05:22:27.922111+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849865]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:21:50
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849865/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849865/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1ed0cd2e20304bdd961e3f7fc0d9772b

    "2022-05-27T05:19:08.482798+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849866]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only}
-----------------------------------------------------------------
time:   00:15:46
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849866/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849866/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8309c1e2f4494a748ae0c508bbef9db6

    "2022-05-27T05:18:32.237028+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849867]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:46
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849867/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849867/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe6f0e67bf574da4bdab6b5942c50258

    "2022-05-27T05:22:19.718096+0000 mon.a (mon.0) 163 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849868]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:17:49
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849868/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849868/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe8d0210b1504d298e81d38b21fb4fff

    "2022-05-27T05:22:11.673271+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849869]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:18:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849869/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849869/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ddc686f2ad24c2bb4fb62cbbf8a4b33

    "2022-05-27T05:22:32.773196+0000 mon.a (mon.0) 109 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849870]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:17:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849870/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849870/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c3e5b8f5b5642748eca7c28fa2cdfe4

    "2022-05-27T13:01:41.051644+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849871]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:17:23
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849871/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849871/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54270e911a4e4ea6bde30fa997b5d081

    "2022-05-27T13:05:14.195082+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849872]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:22:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849872/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849872/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ae8df9d79d344f5b3daba270e387293

    "2022-05-27T13:05:47.860457+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849873]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:22:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849873/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849873/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e4daa760ea5640c1887f3d08b996beb8

    "2022-05-27T13:06:56.173626+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849874]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849874/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849874/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13105de96cb544c18e27951a9c97f8b6

    "2022-05-27T13:05:37.067422+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849875]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:16:37
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849875/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849875/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ad3b7ac4ae342109ad5378d7cc9b390

    "2022-05-27T13:07:46.168853+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849876]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:18:09
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849876/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849876/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3cd74eaf9ede4984bc05134bf64e07e5

    "2022-05-27T13:08:09.705412+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849877]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:18:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849877/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849877/
sentry: https://sentry.ceph.com/organizations/ceph/?query=53ceeb817d234f608948570ea26918ba

    "2022-05-27T13:08:16.660967+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849878]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:17:40
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849878/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849878/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e8b7e07e606c42089cac1c55e50942f6

    "2022-05-27T13:08:50.242976+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849879]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:21:53
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849879/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849879/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d82974c2193a45649fc821df0a1118da

    "2022-05-27T13:11:37.955370+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849880]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:17:06
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849880/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849880/
sentry: https://sentry.ceph.com/organizations/ceph/?query=83f8859d2ed04c82ab8a27d4993ae5b4

    "2022-05-27T13:16:05.234010+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849881]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:21:56
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849881/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849881/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b84217335ce8428abe8d6ddd6e0011cd

    "2022-05-27T13:16:52.358678+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849882]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:22:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849882/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849882/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5821d572f4344fe924947d09a7ea47d

    "2022-05-27T13:18:01.449196+0000 mon.a (mon.0) 101 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849883]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten}
-----------------------------------------------------------------
time:   00:16:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849883/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849883/
sentry: https://sentry.ceph.com/organizations/ceph/?query=21d9b372078a4e20b446b30508dbbf94

    "2022-05-27T13:17:08.544069+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849884]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:51
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849884/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849884/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f0c069262fba455cb2360cdd3abce332

    "2022-05-27T13:20:37.648917+0000 mgr.y (mgr.4119) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849885]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:17:29
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849885/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849885/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1ffd5b794d64cc8a934d57dc8f36125

    "2022-05-27T13:20:35.778569+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849886]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:17:27
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849886/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849886/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72c591cb5d9c448cac547085f18c6ec0

    "2022-05-27T13:20:57.032407+0000 mgr.x (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849887]  krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap}
-----------------------------------------------------------------
time:   00:16:12
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849887/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849887/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ff99bd7d8934b1bb48697cac2f72fdf

    "2022-05-27T13:19:31.202214+0000 mon.a (mon.0) 61 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849888]  krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets}
-----------------------------------------------------------------
time:   00:17:04
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849888/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849888/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c73e8bc2b1d747959c1a60326d061396

    "2022-05-27T13:21:28.855323+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849889]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:22:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849889/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849889/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56eed150b1be411596701dba77ce4ca9

    "2022-05-27T13:22:43.224019+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849890]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:22:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849890/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849890/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a3709b7b51a4088aa2846c759a6aed9

    "2022-05-27T13:25:45.044761+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849891]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:16:59
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849891/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849891/
sentry: https://sentry.ceph.com/organizations/ceph/?query=65b527d3130b41ea84215f6e0e273665

    "2022-05-27T13:24:44.790472+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849892]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:22:20
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849892/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c86a355c51a44598bcdb3fe8ad89e3d9

    "2022-05-27T13:26:24.164495+0000 mon.a (mon.0) 150 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849893]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:17:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849893/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849893/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae0a7b56ca03498cbe50717ba0bf134b

    "2022-05-27T13:26:40.290374+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849894]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:17:39
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849894/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849894/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ec3f6d5c36c4977882a524ceccff2e6

    "2022-05-27T13:26:22.431442+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849895]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard}
-----------------------------------------------------------------
time:   00:16:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849895/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849895/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c1fdaea72f2a4636aa291fd426240085

    "2022-05-27T13:25:00.199507+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849896]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849896/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1aac2d06d33944e8a4e3e55023bb20fc

    "2022-05-27T13:27:50.964507+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849897]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:18:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849897/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849897/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e0c03f984267498483a82838e564b1b3

    "2022-05-27T13:29:29.652815+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849898]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:35
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849898/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849898/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0bf157aca1d46fa8fea00ae53b7e56d

    "2022-05-27T13:29:31.202872+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849899]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:17:25
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849899/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849899/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42dac6e1b1e042d5b203bb0a70eefe96

    "2022-05-27T13:32:05.321941+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849900]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:22:23
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849900/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849900/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93f3bd25de584a839635c57c109ff1e0

    "2022-05-27T13:31:04.664593+0000 mon.a (mon.0) 103 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849901]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:17:32
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849901/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849901/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e5385d6fcedb4ee9b8a5c22e3ca4d3a2

    "2022-05-27T13:33:43.294264+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849902]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849902/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e8c17ccff55a41c8a81fe4fb611d7604

    "2022-05-27T13:32:56.140049+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849903]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_huge_image}
-----------------------------------------------------------------
time:   00:16:27
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849903/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849903/
sentry: https://sentry.ceph.com/organizations/ceph/?query=479db99968aa49f39e86a12c998da43e

    "2022-05-27T13:31:41.667633+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849904]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:16:32
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849904/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849904/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a4a719db1b8410682212b019289e95b

    "2022-05-27T13:33:51.601317+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849905]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:53
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849905/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849905/
sentry: https://sentry.ceph.com/organizations/ceph/?query=124e43ff6f7741b4803b2393e1b7e7b5

    "2022-05-27T13:35:04.279835+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849906]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:22:13
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849906/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849906/
sentry: https://sentry.ceph.com/organizations/ceph/?query=798d44a52c824139adb61b301fd51dab

    "2022-05-27T13:34:14.698365+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849907]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:22:17
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849907/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b8adb27f8fa455caff61ba25ca0cb9f

    "2022-05-27T13:36:58.358628+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849908]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:17:57
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849908/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849908/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c09c57ef9d64e24a3549fc64ded462e

    "2022-05-27T13:36:52.388666+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849909]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:17:32
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849909/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849909/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36101198ebe54dfd835f77aca1a33b42

    "2022-05-27T13:36:46.597081+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849910]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:17:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849910/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849910/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b2be5c17c1148bc8c7f4d3d1536c254

    "2022-05-27T13:38:12.833061+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849911]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:17:34
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849911/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849911/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5e08f8b95a1c4260bef591b4160ec4e6

    "2022-05-27T13:40:18.450662+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849912]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_modprobe}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849912/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849912/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1931a74f1cd74cc8b9406dc698729971

    "2022-05-27T13:38:43.870323+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849913]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:17:33
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849913/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22a9ca59f2dc4a60b991e5a452ef3f33

    "2022-05-27T13:41:13.385095+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849914]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:17:31
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849914/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8f9b8a6864a4cf48f5af55b33c73a09

    "2022-05-27T13:42:29.392643+0000 mon.a (mon.0) 119 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849915]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:17:41
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849915/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849915/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b31802f167d24b3bb8e774bf9261ffb4

    "2022-05-27T13:44:46.263631+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849916]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:17:21
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849916/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b56444fe9d74b768d03c80e9c4c5958

    "2022-05-27T13:43:04.341819+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849917]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:22:17
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849917/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6ce908e4322942b094fb19e91884cb21

    "2022-05-27T13:44:37.932795+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849918]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849918/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42aae544145945fbaedb4362fdc7b281

    "2022-05-27T13:45:17.181752+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849919]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:18:21
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849919/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=270a013f89bd498da9496477af49838e

    "2022-05-27T13:48:59.485343+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849920]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments}
-----------------------------------------------------------------
time:   00:15:59
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849920/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2d777492664c4ea6bb23c023c0f55e9d

    "2022-05-27T13:47:19.129180+0000 mon.a (mon.0) 80 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849921]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:30
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849921/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849921/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2eccf1702d2f4bee9b2cc6413026eb61

    "2022-05-27T13:54:59.822045+0000 mon.a (mon.0) 109 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849922]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_fio}
-----------------------------------------------------------------
time:   00:23:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849922/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a2aa93566e034075b10b070f586ccf93

    "2022-05-27T13:54:37.991064+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849923]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_data_pool}
-----------------------------------------------------------------
time:   00:23:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849923/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849923/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba1cd9b34aa44a519f9fdbf52f1c3046

    "2022-05-27T13:54:25.892587+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849924]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option}
-----------------------------------------------------------------
time:   00:18:09
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849924/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8cf08571934445591ebe233325e406e

    "2022-05-27T13:53:56.942338+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849925]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build}
-----------------------------------------------------------------
time:   00:18:06
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849925/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aa10855cbf614eb28d7f15b10911ab1a

    "2022-05-27T13:54:01.348856+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849926]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_fallocate}
-----------------------------------------------------------------
time:   00:18:26
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849926/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849926/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dbfd7de60e644f9b4d1d26f5dc6ae0b

    "2022-05-27T13:55:31.244093+0000 mon.a (mon.0) 94 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849927]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap}
-----------------------------------------------------------------
time:   00:18:28
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849927/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849927/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7a2b8db92884f798bcbbe4532764d17

    "2022-05-27T13:55:30.940318+0000 mon.a (mon.0) 107 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849928]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench}
-----------------------------------------------------------------
time:   00:22:56
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849928/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=257c4fdea841453bbb964180d0dd0e05

    "2022-05-27T13:55:17.661878+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849929]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map}
-----------------------------------------------------------------
time:   00:17:42
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849929/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849929/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c9efa1753bf4021979f1c08cf54324a

    "2022-05-27T13:56:50.851338+0000 mgr.x (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849930]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_parent_overlap}
-----------------------------------------------------------------
time:   00:15:48
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849930/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849930/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c5b6e2a77b14807ae7b173df93c8ae4

    "2022-05-27T13:55:02.707800+0000 mon.a (mon.0) 83 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849931]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:17:57
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849931/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849931/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e408eb2b1a054ef0bea04e7f507bef0f

    "2022-05-27T13:57:37.317307+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849932]  krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/upmap thrashosds-health workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:18:39
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849932/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849932/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2f582c75b904b5397e70f406db4d502

    "2022-05-27T13:58:04.440491+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849933]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_namespaces}
-----------------------------------------------------------------
time:   00:18:10
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849933/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849933/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95ffa60651694a96b1adc86251775c9f

    "2022-05-27T13:57:59.782584+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849934]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:17:55
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849934/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849934/
sentry: https://sentry.ceph.com/organizations/ceph/?query=441df46118104590bdef888e236a1d87

    "2022-05-27T13:57:52.368913+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849935]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate}
-----------------------------------------------------------------
time:   00:22:45
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849935/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849935/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ddec13c62cc43a7bc78d0f967422222

    "2022-05-27T14:03:13.982196+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849936]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs}
-----------------------------------------------------------------
time:   00:17:36
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849936/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4cb76bbb6913486782fd8bb32aacb6f5

    "2022-05-27T14:00:53.011662+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849937]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:17:54
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849937/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849937/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6b4beb3ec36d489da5c8068275035193

    "2022-05-27T14:03:37.275441+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849938]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netns}
-----------------------------------------------------------------
time:   00:18:00
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849938/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849938/
sentry: https://sentry.ceph.com/organizations/ceph/?query=92521a12e2f449b2bd2b2747372edc17

    "2022-05-27T14:02:57.348433+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849939]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_read_only}
-----------------------------------------------------------------
time:   00:17:02
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849939/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849939/
sentry: https://sentry.ceph.com/organizations/ceph/?query=208df42da70744eabbede20382c31b58

    "2022-05-27T13:57:39.899125+0000 mon.a (mon.0) 79 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849940]  krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client}
-----------------------------------------------------------------
time:   00:17:41
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849940/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849940/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cf0560f0b6b432f85d4c8a939e4593b

    "2022-05-27T14:05:51.455332+0000 mgr.x (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849941]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks}
-----------------------------------------------------------------
time:   00:22:35
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849941/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849941/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8af7f604aadd4addba954ca009a9644f

    "2022-05-27T14:04:42.852697+0000 mgr.y (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849942]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4}
-----------------------------------------------------------------
time:   00:18:06
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849942/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849942/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19a2ba67c3d84f729cc678967f0b1ffd

    "2022-05-27T14:06:35.811511+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849943]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent}
-----------------------------------------------------------------
time:   00:23:24
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849943/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849943/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7033233ff674c4cb7b86a29e3ef6bd1

    "2022-05-27T14:06:23.347498+0000 mon.a (mon.0) 91 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849944]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_huge_tickets}
-----------------------------------------------------------------
time:   00:18:05
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849944/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849944/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d92fe62e4c84dc998db1f71ec192dc0

    "2022-05-27T14:06:30.824747+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849945]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:18:01
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849945/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b3ee0b0e64aa403a8767f368f4ab3d1a

    "2022-05-27T14:08:42.650380+0000 mon.a (mon.0) 94 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849946]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_image_read}
-----------------------------------------------------------------
time:   00:23:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849946/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849946/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71663ac83b3441b69133a5fedc33ed25

    "2022-05-27T14:08:23.392382+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849947]  krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout}
-----------------------------------------------------------------
time:   00:16:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849947/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849947/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c4a7912c70d243acad643cea7b237c3a

    "2022-05-27T14:06:34.426835+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849948]  krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client}
-----------------------------------------------------------------
time:   00:17:50
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849948/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849948/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a389fc2e29964466a119515d6f1462b4

    "2022-05-27T14:09:11.553462+0000 mon.a (mon.0) 185 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849949]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel}
-----------------------------------------------------------------
time:   00:17:25
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849949/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849949/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2eb1e8c076524f86a56a9fe7a0ad2d25

    "2022-05-27T14:09:42.072199+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849950]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone}
-----------------------------------------------------------------
time:   00:17:09
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849950/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849950/
sentry: https://sentry.ceph.com/organizations/ceph/?query=92c730b8a8fc466392a22281ba81734e

    "2022-05-27T14:10:50.465864+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849951]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_snapshot_io}
-----------------------------------------------------------------
time:   00:17:15
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849951/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c06c2e43ffd41fdbe30dcef5b07603b

    "2022-05-27T14:11:27.863470+0000 mgr.x (mgr.4126) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849952]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_unmap}
-----------------------------------------------------------------
time:   00:17:22
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849952/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849952/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e5952014e194920b4511e77a25c4890

    "2022-05-27T14:11:55.110263+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log

[6849953]  krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync}
-----------------------------------------------------------------
time:   00:17:08
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849953/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849953/
sentry: https://sentry.ceph.com/organizations/ceph/?query=abc5c99101c04dafa240401fb3964a1f

    "2022-05-27T14:16:48.307657+0000 mon.a (mon.0) 102 : cluster [WRN] Health
    check failed: 3 mgr modules have failed dependencies
    (MGR_MODULE_DEPENDENCY)" in cluster log

[6849954]  krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big}
-----------------------------------------------------------------
time:   00:17:12
info:   http://pulpito.ceph.com/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849954/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-27_03:20:03-krbd-quincy-testing-default-smithi/6849954/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5ebf346863a496e9d250ba20359b0b7

    "2022-05-27T14:17:29.811572+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes,
    test_orchestrator" in cluster log