2023-08-15T23:11:40.526 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-15T23:11:40.527 INFO:teuthology.results:subset: '111/120000'
2023-08-15T23:11:40.527 INFO:teuthology.results:seed: '9299'
2023-08-16T01:17:10.562 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-16T01:17:10.588 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-16T12:36:30.337 INFO:teuthology.results:Tests finished! gathering results...
2023-08-16T12:36:45.861 INFO:scrape:Found 180 jobs
2023-08-16T12:36:46.704 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369161/teuthology.log
2023-08-16T12:37:05.363 INFO:scrape:Found 9 distinct failure reasons
2023-08-16T12:37:05.365 INFO:scrape:Failure: "2023-08-16T01:05:10.050071+0000 mon.a (mon.0) 52 : cluster [WRN] Health check failed: Module 'devicehealth' has failed dependency: cannot import name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log
2023-08-16T12:37:05.365 INFO:scrape:146 jobs
2023-08-16T12:37:05.366 INFO:scrape:suites intersection: []
2023-08-16T12:37:05.367 INFO:scrape:suites union: ['2-cluster/{fix-2', '2-clusters/{fixed-3', '3-objectstore/bluestore-bitmap', '3-objectstore/bluestore-comp-lz4', '3-objectstore/bluestore-comp-snappy', '3-objectstore/bluestore-comp-zlib', '3-objectstore/bluestore-comp-zstd', '3-objectstore/bluestore-low-osd-mem-target', '3-objectstore/bluestore-stupid', '3-supported-random-distro$/{centos_8}', '3-supported-random-distro$/{rhel_8}', '4-cache-path', '4-supported-random-distro$/{centos_8}', '4-supported-random-distro$/{rhel_8}', '4-supported-random-distro$/{ubuntu_latest}', '5-cache-mode/rwl', '5-cache-mode/ssd', '5-pool/ec-data-pool', '5-pool/none', '5-pool/replicated-data-pool', '6-cache-size/1G', '6-cache-size/5G', '6-cache-size/8G', '6-prepare/qcow2-file', '6-prepare/qcow2-http', '6-prepare/raw-file', '7-io-workloads/qemu_xfstests', '7-workloads/c_api_tests_with_defaults}', '7-workloads/fio}', '7-workloads/qemu_xfstests}', '7-workloads/recovery}', '8-migrate-workloads/execute', '9-cleanup/cleanup}', 'cachepool/none', 'cachepool/small', 'centos_latest', 'clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'cluster/{fixed-3', 'clusters/{fixed-1', 'clusters/{fixed-2', 'clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'config/permit-partial-discard', 'features/defaults', 'features/format-1', 'features/journaling', 'features/layering', 'features/readbalance', 'min-compat-client/default', 'min-compat-client/octopus', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack', 'openstack}', 'policy/none', 'policy/simple', 'pool/ceph_and_immutable_object_cache', 'pool/ec-cache-pool', 'pool/ec-data-pool', 'pool/none', 'pool/replicated-data-pool', 'pool/small-cache-pool', 'qemu/xfstests', 'rbd-mirror/four-per-cluster', 'rbd/basic/{base/install', 'rbd/cli/{base/install', 'rbd/cli_v1/{base/install', 'rbd/encryption/{cache/none', 'rbd/encryption/{cache/writearound', 'rbd/encryption/{cache/writeback', 'rbd/encryption/{cache/writethrough', 'rbd/immutable-object-cache/{clusters/{fix-2', 'rbd/librbd/{cache/none', 'rbd/librbd/{cache/writearound', 'rbd/librbd/{cache/writeback', 'rbd/librbd/{cache/writethrough', 'rbd/maintenance/{base/install', 'rbd/migration/{1-base/install', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'rbd/nbd/{base/install', 'rbd/pwl-cache/home/{1-base/install', 'rbd/pwl-cache/tmpfs/{1-base/install', 'rbd/qemu/{cache/none', 'rbd/qemu/{cache/writearound', 'rbd/qemu/{cache/writeback', 'rbd/qemu/{cache/writethrough', 'rbd/singleton-bluestore/{all/issue-20295', 'rbd/singleton/{all/admin_socket', 'rbd/singleton/{all/formatted-output', 'rbd/singleton/{all/merge_diff', 'rbd/singleton/{all/permissions', 'rbd/singleton/{all/qemu-iotests-no-cache', 'rbd/singleton/{all/qemu-iotests-writearound', 'rbd/singleton/{all/qemu-iotests-writethrough', 'rbd/singleton/{all/qos', 'rbd/singleton/{all/rbd-vs-unmanaged-snaps', 'rbd/singleton/{all/rbd_mirror', 'rbd/singleton/{all/rbd_tasks', 'rbd/singleton/{all/read-flags-no-cache', 'rbd/singleton/{all/read-flags-writeback', 'rbd/singleton/{all/read-flags-writethrough', 'rbd/singleton/{all/snap-diff', 'rbd/thrash/{base/install', 'rbd/valgrind/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/rbd_python_api_tests_old_format}', 'thrashers/cache', 'thrashers/default', 'thrashosds-health', 'validator/memcheck', 'workloads/c_api_tests_with_defaults}', 'workloads/c_api_tests_with_journaling}', 'workloads/c_api_tests}', 'workloads/dynamic_features_no_cache}', 'workloads/dynamic_features}', 'workloads/fio_on_immutable_object_cache}', 'workloads/fsx}', 'workloads/journal}', 'workloads/python_api_tests_with_defaults}', 'workloads/python_api_tests_with_journaling}', 'workloads/python_api_tests}', 'workloads/qemu_bonnie}', 'workloads/qemu_fsstress}', 'workloads/qemu_on_immutable_object_cache_and_thrash}', 'workloads/qemu_xfstests_luks1}', 'workloads/qemu_xfstests_luks2}', 'workloads/qemu_xfstests}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-ha-workunit}', 'workloads/rbd-mirror-journal-bootstrap-workunit}', 'workloads/rbd-mirror-journal-stress-workunit}', 'workloads/rbd-mirror-journal-workunit}', 'workloads/rbd-mirror-snapshot-bootstrap-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-workunit-journaling}', 'workloads/rbd-mirror-snapshot-workunit-minimum}', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'workloads/rbd-mirror-workunit-config-key}', 'workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'workloads/rbd-mirror-workunit-policy-none}', 'workloads/rbd_api_tests_copy_on_read}', 'workloads/rbd_api_tests_journaling}', 'workloads/rbd_api_tests_no_locking}', 'workloads/rbd_api_tests}', 'workloads/rbd_cli_generic}', 'workloads/rbd_cli_groups}', 'workloads/rbd_cli_import_export}', 'workloads/rbd_cli_luks_encryption}', 'workloads/rbd_cli_migration}', 'workloads/rbd_fio}', 'workloads/rbd_fsx_cache_writearound}', 'workloads/rbd_fsx_cache_writeback}', 'workloads/rbd_fsx_cache_writethrough}', 'workloads/rbd_fsx_copy_on_read}', 'workloads/rbd_fsx_deep_copy}', 'workloads/rbd_fsx_journal}', 'workloads/rbd_fsx_nbd}', 'workloads/rbd_fsx_nocache}', 'workloads/rbd_mirror}', 'workloads/rbd_nbd_diff_continuous}', 'workloads/rbd_nbd}', 'workloads/rebuild_object_map}']
2023-08-16T12:37:05.367 INFO:scrape:
2023-08-16T12:37:05.368 INFO:scrape:Dead: None
2023-08-16T12:37:05.368 INFO:scrape:20 jobs: ['7369161', '7369021', '7369022', '7369023', '7369153', '7369096', '7369041', '7369037', '7369029', '7369010', '7369047', '7369154', '7369152', '7369026', '7369032', '7369031', '7369030', '7369025', '7368996', '7369051']
2023-08-16T12:37:05.368 INFO:scrape:
2023-08-16T12:37:05.369 INFO:scrape:Failure: Command failed on smithi131 with status 1: 'install -d -m0755 -- /home/ubuntu/cephtest/ceph.data'
2023-08-16T12:37:05.369 INFO:scrape:1 jobs: ['7369013']
2023-08-16T12:37:05.369 INFO:scrape:suites: ['clusters/{fixed-3', 'objectstore/bluestore-comp-zstd', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/dynamic_features}']
2023-08-16T12:37:05.370 INFO:scrape:
2023-08-16T12:37:05.370 INFO:scrape:Failure: ['/dev/vg_nvme: already exists in filesystem', "Creating volume group 'vg_nvme' failed", "Run `vgcreate --help' for more information."]
2023-08-16T12:37:05.370 INFO:scrape:5 jobs: ['7369028', '7369035', '7369034', '7369033', '7369020']
2023-08-16T12:37:05.370 INFO:scrape:suites intersection: []
2023-08-16T12:37:05.371 INFO:scrape:suites union: ['centos_latest', 'cluster/{fixed-3', 'clusters/{fixed-1', 'clusters/{fixed-3', 'features/journaling', 'features/readbalance', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack', 'openstack}', 'pool/datapool', 'pool/ec-data-pool', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'rbd/iscsi/{base/install', 'rbd/qemu/{cache/writethrough', 'rbd/singleton/{all/verify_pool', 'rbd/valgrind/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'validator/memcheck', 'workloads/ceph_iscsi}', 'workloads/python_api_tests_with_journaling}', 'workloads/qemu_fsstress}', 'workloads/rbd_cli_groups}']
2023-08-16T12:37:05.371 INFO:scrape:
2023-08-16T12:37:05.371 INFO:scrape:Failure: Stale jobs detected, aborting.
2023-08-16T12:37:05.372 INFO:scrape:3 jobs: ['7369019', '7369083', '7369129']
2023-08-16T12:37:05.372 INFO:scrape:suites intersection: ['openstack}']
2023-08-16T12:37:05.372 INFO:scrape:suites union: ['2-clusters/{fixed-3', '3-objectstore/bluestore-hybrid', '4-supported-random-distro$/{rhel_8}', '5-pool/ec-data-pool', '6-prepare/qcow2-file', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup}', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/ceph_and_immutable_object_cache', 'rbd/immutable-object-cache/{clusters/{fix-2', 'rbd/migration/{1-base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/c_api_tests_with_defaults}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}']
2023-08-16T12:37:05.373 INFO:scrape:
2023-08-16T12:37:05.373 INFO:scrape:Dead: b'2023-08-16T12:33:28.515 DEBUG:teuthology.exit:Finished running handlers'
2023-08-16T12:37:05.373 INFO:scrape:1 jobs: ['7369089']
2023-08-16T12:37:05.374 INFO:scrape:suites: ['clusters/{fixed-3', 'features/readbalance', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/encryption/{cache/writeback', 'supported-random-distro$/{centos_8}', 'workloads/qemu_xfstests_luks1}']
2023-08-16T12:37:05.374 INFO:scrape:
2023-08-16T12:37:05.374 INFO:scrape:Failure: SSH connection to smithi159 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'
2023-08-16T12:37:05.374 INFO:scrape:2 jobs: ['7368998', '7369149']
2023-08-16T12:37:05.375 INFO:scrape:suites intersection: []
2023-08-16T12:37:05.375 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '6-cache-size/5G', '7-workloads/qemu_xfstests}', 'objectstore/bluestore-comp-zstd', 'openstack', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install', 'rbd/singleton/{all/rbdmap_RBDMAPFILE', 'supported-random-distro$/{rhel_8}}']
2023-08-16T12:37:05.375 INFO:scrape:
2023-08-16T12:37:05.376 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.159
2023-08-16T12:37:05.376 INFO:scrape:1 jobs: ['7369003']
2023-08-16T12:37:05.376 INFO:scrape:suites: ['2-clusters/{fixed-3', '3-objectstore/bluestore-stupid', '4-supported-random-distro$/{centos_8}', '5-pool/replicated-data-pool', '6-prepare/qcow2-file', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup}', 'openstack}', 'rbd/migration/{1-base/install']
2023-08-16T12:37:05.377 INFO:scrape:
2023-08-16T12:37:05.377 INFO:scrape:Failure: ['Failed to connect to the host via ssh: ssh: connect to host smithi159.front.sepia.ceph.com port 22: No route to host']
2023-08-16T12:37:05.377 INFO:scrape:1 jobs: ['7369124']
2023-08-16T12:37:05.378 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}']
2023-08-16T12:37:05.378 INFO:scrape:
2023-08-16T12:37:05.384 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/
failed:  153
dead:    27
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7368990]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:17:38
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368990/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368990/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc48a7c9b8ce49909a0e9c815bdc6ada

    "2023-08-15T23:34:12.446869+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7368991]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:14:50
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368991/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368991/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2913a64a3c3c4bf9b2a71c49a763e774

    "2023-08-15T23:32:47.119274+0000 mon.a (mon.0) 66 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7368992]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}
-----------------------------------------------------------------
time:   00:22:06
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368992/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368992/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5aff98c5cda7450f81f0358411d7df6d

    "2023-08-15T23:37:56.139324+0000 mon.a (mon.0) 79 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7368993]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy}
-----------------------------------------------------------------
time:   00:17:49
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368993/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368993/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1849041c23e3419887d3c88dec6cd9cc

    "2023-08-15T23:39:06.071832+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7368994]  rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:20:42
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368994/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368994/
sentry: https://sentry.ceph.com/organizations/ceph/?query=63fc51f6b38b46cf9e02b8884eda7eb8

    "2023-08-15T23:49:01.954737+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7368995]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_migration}
-----------------------------------------------------------------
time:   00:17:23
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368995/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368995/
sentry: https://sentry.ceph.com/organizations/ceph/?query=631b7cce6d7f430abb96e2795103df1b

    "2023-08-15T23:44:35.025454+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7368997]  rbd/singleton/{all/read-flags-no-cache objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:19:02
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368997/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368997/
sentry: https://sentry.ceph.com/organizations/ceph/?query=238eae1708f147b2a9e9927093e437eb

    "2023-08-15T23:44:58.641114+0000 mon.a (mon.0) 51 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7368999]  rbd/encryption/{cache/writearound clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_xfstests_luks2}
-----------------------------------------------------------------
time:   00:33:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368999/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368999/
sentry: https://sentry.ceph.com/organizations/ceph/?query=68900e30316d4d4788fb529fb151e3dc

    "2023-08-16T00:00:38.044657+0000 mon.a (mon.0) 150 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369000]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zlib qemu/xfstests supported-random-distro$/{centos_8} workloads/rebuild_object_map}
-----------------------------------------------------------------
time:   00:18:33
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369000/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369000/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d332fbdcbc6345ca89b52d0867f6b3f0

    "2023-08-15T23:51:15.986376+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369001]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}
-----------------------------------------------------------------
time:   00:14:40
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369001/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369001/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f8a2a1d9b184206a71c460ce7cb76ca

    "2023-08-15T23:46:23.078488+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369002]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_journal}
-----------------------------------------------------------------
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369002/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369002/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d0fd28782bf848c18640da4220c69697

    "2023-08-15T23:50:44.302361+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369003]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-stupid 4-supported-random-distro$/{centos_8} 5-pool/replicated-data-pool 6-prepare/qcow2-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369003/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369003/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db88c2033d0b4450a3eaecb30fbd5f2c

    [Errno None] Unable to connect to port 22 on 172.21.15.159

[7369004]  rbd/singleton/{all/read-flags-writeback objectstore/bluestore-stupid openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:21:54
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369004/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369004/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9e7a069dd214032aff80ba26f17de4d

    "2023-08-15T23:56:18.707414+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369005]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:17:21
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369005/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369005/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e25afc2e9764cf190ecac285d591360

    "2023-08-15T23:53:36.177309+0000 mon.a (mon.0) 56 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369006]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}
-----------------------------------------------------------------
time:   00:21:05
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369006/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369006/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba182fef342c48b8b958ccb42934c5c9

    "2023-08-15T23:57:10.932516+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369007]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd}
-----------------------------------------------------------------
time:   00:21:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369007/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369007/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ce95efb8198f4f67b767e53c60e6fb8c

    "2023-08-15T23:57:32.534354+0000 mon.a (mon.0) 82 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369008]  rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/rbd_cls_tests}
-----------------------------------------------------------------
time:   00:19:52
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369008/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369008/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b655c16244814b2fae0bfe236b1596b1

    "2023-08-15T23:55:59.001242+0000 mon.a (mon.0) 69 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369009]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/python_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:31:24
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369009/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369009/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee00e46a19b64e3289ca6e724fce88ff

    "2023-08-16T00:10:03.382078+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369011]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:17:17
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369011/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369011/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ffa879d130e4497ac6fb91cf0ae1969

    "2023-08-15T23:56:22.934000+0000 mon.a (mon.0) 63 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369012]  rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-cache-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_bonnie}
-----------------------------------------------------------------
time:   00:15:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369012/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369012/
sentry: https://sentry.ceph.com/organizations/ceph/?query=189c0ed21fb64ac0810dd243e9f18525

    "2023-08-15T23:57:10.395522+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369013]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features}
-----------------------------------------------------------------
time:   00:16:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369013/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369013/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c337c8a66c14505950fd5e6db594cc1

    Command failed on smithi131 with status 1: 'install -d -m0755 --
    /home/ubuntu/cephtest/ceph.data'

[7369014]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none}
-----------------------------------------------------------------
time:   00:14:42
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369014/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369014/
sentry: https://sentry.ceph.com/organizations/ceph/?query=868a8747960f4dd9a8812f7eb0dc4a2e

    "2023-08-15T23:57:05.266903+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369015]  rbd/singleton/{all/read-flags-writethrough objectstore/filestore-xfs openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:17:31
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369015/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369015/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14cbb965e8e24e7aba7070f29cb16c39

    "2023-08-15T23:59:53.532656+0000 mon.a (mon.0) 47 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369016]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_nocache}
-----------------------------------------------------------------
time:   00:20:45
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369016/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369016/
sentry: https://sentry.ceph.com/organizations/ceph/?query=55b6b4c1f55c435b81b14ed23d401e50

    "2023-08-16T00:06:18.205854+0000 mgr.y (mgr.4119) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369017]  rbd/encryption/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks1}
-----------------------------------------------------------------
time:   00:18:21
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369017/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369017/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44c74fb7c45a43ec851d6afe5595a7d2

    "2023-08-16T00:00:34.549982+0000 mgr.y (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369018]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:19:30
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369018/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369018/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93f9c81385814e06b0a3b87d94e8c651

    "2023-08-16T00:04:27.695211+0000 mon.a (mon.0) 69 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369019]  rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369019/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369019/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab7a59c27b9c4989a2215cc4b211a615

    Stale jobs detected, aborting.

[7369020]  rbd/iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi}
-----------------------------------------------------------------
time:   00:10:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369020/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369020/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22859b0878604b31860207a4a7329777

    ['/dev/vg_nvme: already exists in filesystem', "Creating volume group
    'vg_nvme' failed", 'Failed to connect to the host via ssh: ssh: connect to
    host smithi123.front.sepia.ceph.com port 22: No route to host', "Run
    `vgcreate --help' for more information."]

[7369024]  rbd/singleton/{all/snap-diff objectstore/bluestore-bitmap openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:20:08
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369024/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369024/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad84d24310a041208e89d1ee743e6d65

    "2023-08-16T00:04:44.108782+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369027]  rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/rbd_lock_and_fence}
-----------------------------------------------------------------
time:   00:17:04
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369027/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369027/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13f14e6a05eb46de8e7776b09a8abd65

    "2023-08-16T00:02:53.146525+0000 mon.a (mon.0) 56 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369036]  rbd/encryption/{cache/writethrough clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests_luks2}
-----------------------------------------------------------------
time:   00:15:08
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369036/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369036/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a3d29518fa9b468395f32e769312e8a8

    "2023-08-16T00:05:58.072593+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369038]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369038/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369038/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71b63a4823ef4c788226698c287e0096

    "2023-08-16T00:07:29.354783+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369039]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-bitmap 4-supported-random-distro$/{rhel_8} 5-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:21:14
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369039/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369039/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75b01decc5154cca910e43dd46c7083a

    "2023-08-16T00:16:11.926450+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369040]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:14:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369040/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369040/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ac8fa23ba5b4a3e86e714f6a0b7f322

    "2023-08-16T00:07:46.752756+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369042]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_nbd_diff_continuous}
-----------------------------------------------------------------
time:   00:16:40
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369042/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369042/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ea1c4e203d0144aab9f32a0bb4d156a0

    "2023-08-16T00:11:50.796950+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369043]  rbd/singleton/{all/admin_socket objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:14:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369043/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369043/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67f3c8f8f6874b6084102e9e47fe3e02

    "2023-08-16T00:08:11.876228+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369044]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/journal}
-----------------------------------------------------------------
time:   00:14:57
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369044/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369044/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8def9f1a6a724c71877c1e4e52adb0e9

    "2023-08-16T00:11:30.412782+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369045]  rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/rbd_python_api_tests_old_format}
-----------------------------------------------------------------
time:   00:14:13
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369045/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369045/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e794f1839da41629fae4a30e3c6acaf

    "2023-08-16T00:08:58.710488+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369046]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-bitmap validator/memcheck workloads/rbd_mirror}
-----------------------------------------------------------------
time:   00:29:57
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369046/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369046/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b4c04c594fa4db5ae2fe7956d021832

    "2023-08-16T00:23:58.625664+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369048]  rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:18:40
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369048/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369048/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f93e0ab788044af6b0396336167f7900

    "2023-08-16T00:16:23.471041+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369049]  rbd/singleton/{all/formatted-output objectstore/bluestore-comp-snappy openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:17:44
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369049/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369049/
sentry: https://sentry.ceph.com/organizations/ceph/?query=901222ea7f64483381448dfc6c739861

    "2023-08-16T00:15:01.124499+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369050]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:20:55
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369050/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369050/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75d74f1e3a9540e1b4697f0b9a05d933

    "2023-08-16T00:15:10.740437+0000 mon.a (mon.0) 69 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369052]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:18:42
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369052/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369052/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a7210f242ce4a478e55faf8caa61d35

    "2023-08-16T00:15:43.141718+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369053]  rbd/encryption/{cache/none clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks1}
-----------------------------------------------------------------
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369053/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369053/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b40a8467fa4c43858723717626bacaa1

    "2023-08-16T00:18:56.986083+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369054]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_api_tests}
-----------------------------------------------------------------
time:   00:20:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369054/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369054/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c73220f1df104145a59b1b13e7581db0

    "2023-08-16T00:20:29.338990+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369055]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-stupid qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features}
-----------------------------------------------------------------
time:   00:21:13
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369055/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369055/
sentry: https://sentry.ceph.com/organizations/ceph/?query=001e2eea8cfe447ebebaaee5d103523b

    "2023-08-16T00:26:12.921358+0000 mgr.y (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369056]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-bootstrap-workunit}
-----------------------------------------------------------------
time:   00:14:31
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369056/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369056/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a66ee9fe82d142bea6c5ad3b76fbfb7e

    "2023-08-16T00:18:14.108156+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369057]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}
-----------------------------------------------------------------
time:   00:14:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369057/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369057/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e6a54a2c354c4dafb3a7836ea44a334a

    "2023-08-16T00:17:57.330165+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369058]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_nbd}
-----------------------------------------------------------------
time:   00:18:35
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369058/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369058/
sentry: https://sentry.ceph.com/organizations/ceph/?query=64292834cf804ad09e698f8e08466c39

    "2023-08-16T00:22:27.670496+0000 mon.a (mon.0) 75 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369059]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-lz4 4-supported-random-distro$/{ubuntu_latest} 5-pool/replicated-data-pool 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:16:03
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369059/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369059/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5cab313fd8d344aaa4d625966e39d5a3

    "2023-08-16T00:22:57.528892+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369060]  rbd/singleton/{all/merge_diff objectstore/bluestore-comp-zlib openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:25:25
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369060/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369060/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f6d78c4c20344aac83fb4a57fca219d3

    "2023-08-16T00:32:01.879459+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369061]  rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/rbd_api_tests_old_format}
-----------------------------------------------------------------
time:   00:24:30
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369061/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369061/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fcb22916e8b24a8b97bf7dec172082b1

    "2023-08-16T00:28:54.469882+0000 mon.a (mon.0) 70 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369062]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/c_api_tests}
-----------------------------------------------------------------
time:   00:29:34
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369062/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369062/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2b376f3dd9042c98fc087fbf6e6d0cb

    "2023-08-16T00:35:05.421875+0000 mon.a (mon.0) 62 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369063]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_api_tests_copy_on_read}
-----------------------------------------------------------------
time:   00:15:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369063/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369063/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5077956487a4011b004a8b85b9a5cbf

    "2023-08-16T00:23:33.426143+0000 mon.a (mon.0) 139 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369064]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:17:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369064/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369064/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35bf5616634d4de2a8f00b2282571389

    "2023-08-16T00:25:41.389532+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369065]  rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/8G 7-workloads/fio}
-----------------------------------------------------------------
time:   00:25:42
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369065/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369065/
sentry: https://sentry.ceph.com/organizations/ceph/?query=afb10513c6494831bdee9a4b60f962dd

    "2023-08-16T00:34:17.384503+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369066]  rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/filestore-xfs pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/fsx}
-----------------------------------------------------------------
time:   00:28:20
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369066/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369066/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a69d7cdfc8024a5f88118acdbaf41af0

    "2023-08-16T00:38:26.441205+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369067]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/filestore-xfs qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   00:21:56
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369067/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369067/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc8f7b4b56a94acab7fc2bb3b6b0fc5d

    "2023-08-16T00:31:12.470389+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369068]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-bootstrap-workunit}
-----------------------------------------------------------------
time:   00:15:22
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369068/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369068/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f63215557d540e38dc1d8cc4a7a5b5f

    "2023-08-16T00:25:24.936333+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369069]  rbd/singleton/{all/permissions objectstore/bluestore-comp-zstd openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:17:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369069/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369069/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bf59a46d2c8a4b26a6207f6167299f09

    "2023-08-16T00:26:39.127663+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369070]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_luks_encryption}
-----------------------------------------------------------------
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369070/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9069077452e94b4e8cd3247777146b76

    "2023-08-16T00:24:35.990505+0000 mon.a (mon.0) 77 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369071]  rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/qemu_bonnie}
-----------------------------------------------------------------
time:   00:24:41
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369071/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eb8ac84df3c44e9b991b5e9eb5521afc

    "2023-08-16T00:33:57.005728+0000 mon.a (mon.0) 96 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369072]  rbd/encryption/{cache/writearound clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests_luks2}
-----------------------------------------------------------------
time:   00:15:27
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369072/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369072/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c124666c66d1437e834b9861561e2469

    "2023-08-16T00:28:20.705971+0000 mgr.y (mgr.4138) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369073]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit}
-----------------------------------------------------------------
time:   00:26:05
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369073/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369073/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b3d69edefd44e838f62833e2f2f5386

    "2023-08-16T00:41:44.171598+0000 mon.a (mon.0) 62 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369074]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd}
-----------------------------------------------------------------
time:   00:21:31
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369074/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369074/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fed90828fc174052afdb976d0db0ec2c

    "2023-08-16T00:35:05.485248+0000 mon.a (mon.0) 81 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369075]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-snappy 4-supported-random-distro$/{rhel_8} 5-pool/ec-data-pool 6-prepare/qcow2-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:28:39
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369075/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369075/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b897b0f4ffe74b6ebcce73d8df8ab5f9

    "2023-08-16T00:45:19.585409+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369076]  rbd/singleton/{all/qemu-iotests-no-cache objectstore/bluestore-hybrid openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:14:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369076/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369076/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ddaa5bad96324bfba11036cf82b44765

    "2023-08-16T00:32:12.925561+0000 mon.a (mon.0) 54 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369077]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_api_tests_journaling}
-----------------------------------------------------------------
time:   00:14:36
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369077/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369077/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f548eef24d64e048fc903ec1363a66a

    "2023-08-16T00:33:12.211436+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369078]  rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/rbd_cls_tests}
-----------------------------------------------------------------
time:   00:17:21
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369078/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369078/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c39b239ed9164b77967bd34d408520b3

    "2023-08-16T00:34:55.401372+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369079]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/c_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:29:28
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369079/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369079/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c83cf8dabe28453292e0dfce635f2179

    "2023-08-16T00:46:52.252884+0000 mon.a (mon.0) 60 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369080]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-zstd pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:14:30
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369080/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369080/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e46f3988f7f427ab9a52b325aaed4ca

    "2023-08-16T00:32:27.986121+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369081]  rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{ubuntu_latest} workloads/fio_on_immutable_object_cache}
-----------------------------------------------------------------
time:   00:14:56
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369081/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369081/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4254ab2d928d4847a6328e3885cadffb

    "2023-08-16T00:33:34.984172+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369082]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-bitmap qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   00:23:39
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369082/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369082/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fa198eef175f42c6bd9f9f84a4996fe5

    "2023-08-16T00:43:13.042980+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369083]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369083/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369083/
sentry: https://sentry.ceph.com/organizations/ceph/?query=005ce274746d46efbad364cdd769cca4

    Stale jobs detected, aborting.

[7369084]  rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{ubuntu_latest} workloads/python_api_tests}
-----------------------------------------------------------------
time:   00:15:58
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369084/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369084/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07ef7afc700e4eb7b8f0a818c090d505

    "2023-08-16T00:40:44.689269+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369085]  rbd/singleton/{all/qemu-iotests-writearound objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:18:52
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369085/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369085/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aa724c939ac54ddb97792e1f29835d7a

    "2023-08-16T00:42:10.192030+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369086]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration}
-----------------------------------------------------------------
time:   00:14:03
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369086/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369086/
sentry: https://sentry.ceph.com/organizations/ceph/?query=10b57694af8141d7821be4a17921a9e6

    "2023-08-16T00:39:08.516412+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369087]  rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_fsstress}
-----------------------------------------------------------------
time:   00:30:01
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369087/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369087/
sentry: https://sentry.ceph.com/organizations/ceph/?query=717200aa61ca4f648e9aeb64c235ceb7

    "2023-08-16T00:54:59.723574+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369088]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:20:18
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369088/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369088/
sentry: https://sentry.ceph.com/organizations/ceph/?query=25b3742468e14ddb9ebe95c25382f0e1

    "2023-08-16T00:46:39.337469+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369090]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_api_tests_no_locking}
-----------------------------------------------------------------
time:   00:18:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369090/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1bb44a1259f4872be25341dd3e8456d

    "2023-08-16T00:50:23.527712+0000 mon.a (mon.0) 96 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369091]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:22:55
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369091/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369091/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d39aefb13d748f597d219a921a8a743

    "2023-08-16T00:51:35.193489+0000 mon.a (mon.0) 72 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369092]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}
-----------------------------------------------------------------
time:   00:16:51
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369092/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369092/
sentry: https://sentry.ceph.com/organizations/ceph/?query=459bf6b4dfbd4ff4bab96e241a5d84a0

    "2023-08-16T00:48:34.020125+0000 mon.a (mon.0) 77 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369093]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-lz4 qemu/xfstests supported-random-distro$/{centos_8} workloads/rebuild_object_map}
-----------------------------------------------------------------
time:   00:18:50
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369093/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369093/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e4b7b108935470ebf912e6daf3dfdd5

    "2023-08-16T00:56:42.653310+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369094]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zlib 4-supported-random-distro$/{centos_8} 5-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:23:02
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369094/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369094/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d877e8fc12ba4fffb606350df8cf4ad1

    "2023-08-16T00:57:26.365346+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369095]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff}
-----------------------------------------------------------------
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369095/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369095/
sentry: https://sentry.ceph.com/organizations/ceph/?query=305a3f653bca4718a8346f6924f003af

    "2023-08-16T00:49:33.584440+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369097]  rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/rbd_lock_and_fence}
-----------------------------------------------------------------
time:   00:17:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369097/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369097/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b2a5f1a8a164491a8b28bf40296b6a4

    "2023-08-16T00:49:20.430505+0000 mon.a (mon.0) 75 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369098]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zlib validator/memcheck workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:30:12
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369098/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369098/
sentry: https://sentry.ceph.com/organizations/ceph/?query=82c66b154db745b6b06ec8b0788c15a0

    "2023-08-16T01:03:59.249098+0000 mon.a (mon.0) 60 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369099]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:19:08
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369099/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369099/
sentry: https://sentry.ceph.com/organizations/ceph/?query=00827ce259404c85a3adab162c2f0915

    "2023-08-16T00:52:15.018521+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369100]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/python_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:18:36
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369100/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369100/
sentry: https://sentry.ceph.com/organizations/ceph/?query=211fb2889b8347a1b47769691ac83ade

    "2023-08-16T00:56:06.238367+0000 mon.a (mon.0) 144 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369101]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writearound}
-----------------------------------------------------------------
time:   00:21:04
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369101/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369101/
sentry: https://sentry.ceph.com/organizations/ceph/?query=23e9a155fa9e491198b39b0fce333364

    "2023-08-16T00:58:12.496228+0000 mon.a (mon.0) 94 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369102]  rbd/singleton/{all/qemu-iotests-writethrough objectstore/filestore-xfs openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:16:50
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369102/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369102/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2e467a04d8a44bab5e23e193dcf0101

    "2023-08-16T00:52:09.139915+0000 mon.a (mon.0) 47 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369103]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:14:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369103/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369103/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0e9d5e327e340338a02acbce3d0ef22

    "2023-08-16T00:51:51.880863+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369104]  rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/filestore-xfs pool/ec-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:23:52
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369104/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369104/
sentry: https://sentry.ceph.com/organizations/ceph/?query=82fba2b289724a3f85034b9947afd1aa

    "2023-08-16T01:03:14.653831+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369105]  rbd/encryption/{cache/writethrough clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/filestore-xfs pool/ec-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks2}
-----------------------------------------------------------------
time:   00:22:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369105/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369105/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c838aa4ccc2457fb5da1fb4a021379a

    "2023-08-16T01:02:43.070996+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369106]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-snappy qemu/xfstests supported-random-distro$/{centos_8} workloads/dynamic_features}
-----------------------------------------------------------------
time:   00:22:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369106/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369106/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30112099352340b1989c1e2d997479c6

    "2023-08-16T01:06:37.296939+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369107]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-journaling}
-----------------------------------------------------------------
time:   00:17:51
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369107/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369107/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e8deae636424200a09c18d839978bb4

    "2023-08-16T01:01:05.304545+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369108]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}
-----------------------------------------------------------------
time:   00:14:30
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369108/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369108/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30499018fb214b4d9cfad6b65c66f84b

    "2023-08-16T00:56:12.481498+0000 mon.a (mon.0) 70 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369109]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_nbd}
-----------------------------------------------------------------
time:   00:28:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369109/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369109/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cf6439b244d942fbae1e0518407e4482

    "2023-08-16T01:09:58.412771+0000 mon.a (mon.0) 76 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369110]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:17:24
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369110/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369110/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0cd0d5a7e9ec451ebf52f45acaf3ca30

    "2023-08-16T00:59:36.385504+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369111]  rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_8} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/recovery}
-----------------------------------------------------------------
time:   00:24:25
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369111/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369111/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4fd508f57c547bb97fba5c9dbfe943a

    "2023-08-16T01:08:32.111900+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369112]  rbd/singleton-bluestore/{all/issue-20295 objectstore/bluestore-comp-snappy openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:16:49
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369112/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369112/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4acd56a82d16499595f2d7feaccbc12d

    "2023-08-16T01:05:25.885842+0000 mgr.z (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369113]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zstd 4-supported-random-distro$/{centos_8} 5-pool/replicated-data-pool 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:19:17
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369113/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369113/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9c78afbf26540c2aa768646712faaca

    "2023-08-16T01:07:41.830296+0000 mgr.x (mgr.4122) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369114]  rbd/singleton/{all/qos objectstore/bluestore-bitmap openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:19:31
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369114/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369114/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e2bc412db9f4447960b7dc6ca2d138c

    "2023-08-16T01:05:10.050071+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369115]  rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/rbd_python_api_tests_old_format}
-----------------------------------------------------------------
time:   00:14:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369115/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369115/
sentry: https://sentry.ceph.com/organizations/ceph/?query=946b3ad47e9443808cd735b264de1bd5

    "2023-08-16T01:01:31.786440+0000 mon.a (mon.0) 78 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369116]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zstd validator/memcheck workloads/fsx}
-----------------------------------------------------------------
time:   00:29:39
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369116/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369116/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f78e837afb4348d0bf7411c3936bfbe2

    "2023-08-16T01:16:07.127052+0000 mon.a (mon.0) 70 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369117]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writeback}
-----------------------------------------------------------------
time:   00:18:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369117/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369117/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06d7d92fc19a47f0a50146d7dae65286

    "2023-08-16T01:06:31.089492+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369118]  rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/python_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369118/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369118/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cadd168030a1409c80a2a4f6ce424c09

    "2023-08-16T01:05:35.751734+0000 mgr.x (mgr.4125) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369119]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zlib qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   00:15:47
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369119/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f416a72ac5f460cb309c112bd596107

    "2023-08-16T01:05:15.294772+0000 mgr.y (mgr.4138) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369120]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum}
-----------------------------------------------------------------
time:   00:15:23
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369120/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369120/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f033f54ac75c4e16a3c1cdc637ab7b83

    "2023-08-16T01:05:57.468711+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369121]  rbd/singleton/{all/rbd-vs-unmanaged-snaps objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:14:13
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369121/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369121/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1fa741a4dc4648e7ab4753e310612b70

    "2023-08-16T01:03:37.583472+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369122]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_groups}
-----------------------------------------------------------------
time:   00:17:48
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369122/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369122/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3a9549f6ae7a40e198197507db8c4f1d

    "2023-08-16T01:05:45.634806+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369123]  rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{centos_8} workloads/qemu_bonnie}
-----------------------------------------------------------------
time:   00:18:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369123/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369123/
sentry: https://sentry.ceph.com/organizations/ceph/?query=944a68a2c6054d1fa5b5d94cd1975a3b

    "2023-08-16T01:14:22.953450+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369124]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum}
-----------------------------------------------------------------
time:   00:10:13
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369124/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369124/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4f15ac1d55440188276db1deb220ec4

    ['Failed to connect to the host via ssh: ssh: connect to host
    smithi159.front.sepia.ceph.com port 22: No route to host']

[7369125]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd}
-----------------------------------------------------------------
time:   00:24:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369125/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369125/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f4ff7dbec214711b93ff66797dc76a8

    "2023-08-16T01:18:25.050592+0000 mon.a (mon.0) 78 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369126]  rbd/encryption/{cache/none clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/qemu_xfstests_luks1}
-----------------------------------------------------------------
time:   00:34:25
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369126/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369126/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75144b745ec24d2196b6c77bbcf865da

    "2023-08-16T01:21:09.870818+0000 mgr.x (mgr.4122) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369127]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:26:29
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369127/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369127/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3dd61a63bbf04cb3a2b2a9a3b6685b15

    "2023-08-16T01:19:50.447028+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369128]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writethrough}
-----------------------------------------------------------------
time:   00:18:19
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369128/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369128/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a33c75e8a3fe43a29b00a41714a59063

    "2023-08-16T01:14:07.265380+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369129]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-hybrid 4-supported-random-distro$/{rhel_8} 5-pool/ec-data-pool 6-prepare/qcow2-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369129/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369129/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a417aa2a7c234a24beb45cefe55c98de

    Stale jobs detected, aborting.

[7369130]  rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-snappy openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369130/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369130/
sentry: https://sentry.ceph.com/organizations/ceph/?query=148d3910be924e5686b5a724800b06e2

    "2023-08-16T01:17:27.348671+0000 mon.a (mon.0) 51 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369131]  rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/rbd_api_tests_old_format}
-----------------------------------------------------------------
time:   00:24:54
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369131/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369131/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e92da0f4b5b9404d86016206ae9c58da

    "2023-08-16T01:21:50.445074+0000 mon.a (mon.0) 62 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369132]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-hybrid validator/memcheck workloads/python_api_tests}
-----------------------------------------------------------------
time:   00:29:52
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369132/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369132/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d677ec8805a74316851b8414c84849fa

    "2023-08-16T01:28:16.037594+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369133]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/1G 7-workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:29:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369133/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369133/
sentry: https://sentry.ceph.com/organizations/ceph/?query=49f0c51f54b0464b986c30cea712b6f6

    "2023-08-16T01:26:35.393623+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369134]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-comp-zstd qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map}
-----------------------------------------------------------------
time:   00:15:27
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369134/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369134/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b8e675db24e412cb6d80b89a5b0a293

    "2023-08-16T01:17:54.702891+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369135]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}
-----------------------------------------------------------------
time:   00:20:24
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369135/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369135/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5577e1b0a7a74393a88e7c9b91e49da4

    "2023-08-16T01:22:02.482287+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369136]  rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_fio}
-----------------------------------------------------------------
time:   00:22:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369136/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369136/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f5788777f63b42e2bf1eceb3302be88f

    "2023-08-16T01:24:23.652003+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369137]  rbd/singleton/{all/rbd_tasks objectstore/bluestore-comp-zlib openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:14:32
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369137/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369137/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79aa405f79bb45089a72932fb27e5835

    "2023-08-16T01:19:12.760480+0000 mon.a (mon.0) 54 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369138]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_copy_on_read}
-----------------------------------------------------------------
time:   00:18:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369138/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369138/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5f240c84d31440183ab5a8e5a3eddc4

    "2023-08-16T01:22:18.292528+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369139]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:20:10
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369139/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369139/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81e2336783ce46aeaa3b4a37be990722

    "2023-08-16T01:22:38.925635+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369140]  rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_latest} workloads/qemu_fsstress}
-----------------------------------------------------------------
time:   00:15:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369140/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369140/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4b23275559a4e3b85f0edd73f824868

    "2023-08-16T01:24:23.154577+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369141]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:19:55
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369141/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369141/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ea63c6e29684356829edb8f57781707

    "2023-08-16T01:25:15.486329+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369142]  rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_8} workloads/qemu_on_immutable_object_cache_and_thrash}
-----------------------------------------------------------------
time:   00:20:02
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369142/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369142/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4853d4b9d7e44597a87a8737ec89e2d6

    "2023-08-16T01:28:11.400343+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369143]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}
-----------------------------------------------------------------
time:   00:14:18
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369143/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369143/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e10a8616c90d4779bbf395fa6f75d33c

    "2023-08-16T01:23:17.021105+0000 mon.a (mon.0) 69 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369144]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd}
-----------------------------------------------------------------
time:   00:22:59
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369144/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369144/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97be54dd9cb54337a728e3304c7461a8

    "2023-08-16T01:32:43.889073+0000 mon.a (mon.0) 90 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369145]  rbd/encryption/{cache/writearound clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks2}
-----------------------------------------------------------------
time:   00:21:31
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369145/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e9c0bd7630c4b3eb812c3aa3f0fdb0b

    "2023-08-16T01:32:52.476274+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369146]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-hybrid qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features}
-----------------------------------------------------------------
time:   00:24:44
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369146/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369146/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad7b8eb000f74ecaba57b31d5374be81

    "2023-08-16T01:34:18.865673+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369147]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-low-osd-mem-target 4-supported-random-distro$/{rhel_8} 5-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:25:51
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369147/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da457a998d2046069e08d5303ff21094

    "2023-08-16T01:37:29.622375+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369148]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key}
-----------------------------------------------------------------
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369148/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=186476acae754e8e955bf82bfc202e9e

    "2023-08-16T01:27:52.654069+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369150]  rbd/basic/{base/install cachepool/none clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/rbd_cls_tests}
-----------------------------------------------------------------
time:   00:19:34
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369150/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369150/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae93a455e14149338a633267396f9d0f

    "2023-08-16T01:35:09.434585+0000 mon.a (mon.0) 73 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369151]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-low-osd-mem-target validator/memcheck workloads/python_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:30:53
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369151/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369151/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7dfd1f6dc7d3499581f324fcd5ae0c2e

    "2023-08-16T01:45:05.293629+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369155]  rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/8G 7-workloads/c_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:22:13
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369155/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369155/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9f0eda318e2844cb83819b05038c8bce

    "2023-08-16T01:38:30.517329+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369156]  rbd/singleton/{all/read-flags-no-cache objectstore/bluestore-hybrid openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:20:33
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369156/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369156/
sentry: https://sentry.ceph.com/organizations/ceph/?query=89587a7b9ed946a08c4b536f46701906

    "2023-08-16T01:36:37.221147+0000 mon.a (mon.0) 52 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369157]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-fsx-workunit}
-----------------------------------------------------------------
time:   00:20:23
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369157/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=48fa217e824147d0a1767caeac79c839

    "2023-08-16T01:41:12.382310+0000 mon.a (mon.0) 68 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369158]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}
-----------------------------------------------------------------
time:   00:18:05
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369158/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369158/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36c1b0064ca64871a5c31ca8510956fb

    "2023-08-16T01:34:01.352980+0000 mon.a (mon.0) 74 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369159]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_luks_encryption}
-----------------------------------------------------------------
time:   00:22:37
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369159/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369159/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97844eb8b92b42b69c74db2263e95910

    "2023-08-16T01:40:00.786516+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369160]  rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:21:22
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369160/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369160/
sentry: https://sentry.ceph.com/organizations/ceph/?query=554f51589f6f4fd39d068dee924abe32

    "2023-08-16T01:37:56.678689+0000 mgr.y (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369162]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{rhel_8} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   00:25:15
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369162/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369162/
sentry: https://sentry.ceph.com/organizations/ceph/?query=216b86f120534c199ea55cf63cfa966d

    "2023-08-16T01:45:33.434429+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369163]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-min-compat-client-octopus}
-----------------------------------------------------------------
time:   00:22:30
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369163/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369163/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1fc5055b244e4396b233fe47b1c61cfb

    "2023-08-16T01:39:57.792037+0000 mon.a (mon.0) 71 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369164]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_fsx_journal}
-----------------------------------------------------------------
time:   00:20:45
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369164/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369164/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c90272f64b374467be4179dcab2bb273

    "2023-08-16T01:41:03.893256+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369165]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-stupid 4-supported-random-distro$/{rhel_8} 5-pool/replicated-data-pool 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:22:50
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369165/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369165/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11c62d2846ab472db64905e2b4702ea4

    "2023-08-16T01:40:35.553539+0000 mgr.y (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: devicehealth" in cluster log

[7369166]  rbd/singleton/{all/read-flags-writeback objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:14:46
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369166/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369166/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9de40a8be9a4f1eac76148391150f98

    "2023-08-16T01:34:44.594963+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369167]  rbd/basic/{base/install cachepool/small clusters/{fixed-1 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/rbd_lock_and_fence}
-----------------------------------------------------------------
time:   00:14:39
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369167/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369167/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3aa3267493984408bc47eaa3339c932e

    "2023-08-16T01:34:38.529836+0000 mon.a (mon.0) 70 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' from 'mgr_module' (/usr/share/ceph/mgr/mgr_module.py)
    (MGR_MODULE_DEPENDENCY)" in cluster log

[7369168]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/python_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:29:25
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369168/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369168/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d992998ec1b84688b30d09a93479dfea

    "2023-08-16T01:50:02.166295+0000 mon.a (mon.0) 58 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log

[7369169]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_generic}
-----------------------------------------------------------------
time:   00:18:20
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369169/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c097ea762c85440892c3276415ad0be6

    "2023-08-16T01:38:53.062580+0000 mon.a (mon.0) 62 : cluster [WRN] Health
    check failed: Module 'devicehealth' has failed dependency: cannot import
    name 'MgrDBNotReady' (MGR_MODULE_DEPENDENCY)" in cluster log



Dead
=================================================================
[7368996]  rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368996/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368996/

    Error reimaging machines: Expected smithi159's OS to be rhel 8.4 but found
    ubuntu 20.04

[7368998]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/5G 7-workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368998/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368998/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ff42fa0a1a44dcc876c9ce1b6cc3e5e

    SSH connection to smithi159 was lost: 'sudo DEBIAN_FRONTEND=noninteractive
    apt-get -y install linux-image-generic'

[7369010]  rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{ubuntu_latest} workloads/c_api_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369010/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369010/

    Error reimaging machines: Failed to power on smithi148

[7369021]  rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/c_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369021/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369021/

    Error reimaging machines: Expected smithi159's OS to be rhel 8.4 but found
    centos 8

[7369022]  rbd/singleton-bluestore/{all/issue-20295 objectstore/bluestore-bitmap openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369022/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369022/

    Error reimaging machines: 'wait for sentinel on smithi040' reached maximum
    tries (35) after waiting for 1800 seconds

[7369023]  rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/filestore-xfs 4-supported-random-distro$/{rhel_8} 5-pool/ec-data-pool 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369023/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369023/

    Error reimaging machines: Expected smithi158's OS to be rhel 8.4 but found
    centos 8

[7369025]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369025/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369025/

    Error reimaging machines: Failed to power on smithi157

[7369026]  rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rbd_nbd}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369026/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369026/

    Error reimaging machines: Expected smithi115's OS to be centos 8 but found
    rhel 8.4

[7369028]  rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/filestore-xfs validator/memcheck workloads/python_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:03:26
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369028/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369028/
sentry: https://sentry.ceph.com/organizations/ceph/?query=947f98b9cac047eca9c97bd375e3a5c7

    ['/dev/vg_nvme: already exists in filesystem', "Creating volume group
    'vg_nvme' failed", "Run `vgcreate --help' for more information."]

[7369029]  rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-hybrid qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features_no_cache}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369029/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369029/

    Error reimaging machines: Expected smithi159's OS to be ubuntu 20.04 but
    found centos 8

[7369030]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369030/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369030/

    Error reimaging machines: 500 Server Error: Internal Server Error for url:
    http://fog.front.sepia.ceph.com/fog/host/264/task

[7369031]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_fsx_rate_limit}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369031/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369031/

    Error reimaging machines: Failed to power on smithi188

[7369032]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369032/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369032/

    Error reimaging machines: Expected smithi102's OS to be rhel 8.4 but found
    ubuntu 20.04

[7369033]  rbd/cli/{base/install clusters/{fixed-1 openstack} features/journaling msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_groups}
-----------------------------------------------------------------
time:   00:04:20
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369033/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369033/
sentry: https://sentry.ceph.com/organizations/ceph/?query=519807e3f8a04f1aa8f8e0ebf2324dd2

    ['/dev/vg_nvme: already exists in filesystem', "Creating volume group
    'vg_nvme' failed", "Run `vgcreate --help' for more information."]

[7369034]  rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/qemu_fsstress}
-----------------------------------------------------------------
time:   00:05:12
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369034/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369034/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07a4f6fad76b4f709d352453d1b18af8

    ['/dev/vg_nvme: already exists in filesystem', "Creating volume group
    'vg_nvme' failed", "Run `vgcreate --help' for more information."]

[7369035]  rbd/singleton/{all/verify_pool objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:04:36
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369035/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369035/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86fab91813f649a9a46bf97ecfd2cd38

    ['/dev/vg_nvme: already exists in filesystem', "Creating volume group
    'vg_nvme' failed", "Run `vgcreate --help' for more information."]

[7369037]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369037/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369037/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds

[7369041]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369041/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369041/

    Error reimaging machines: Failed to power on smithi195

[7369047]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369047/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369047/

    Error reimaging machines: Failed to power on smithi105

[7369051]  rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{centos_8} workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369051/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369051/

    Error reimaging machines: Failed to reconnect to smithi123

[7369089]  rbd/encryption/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks1}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369089/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369089/

    hit max job timeout

[7369096]  rbd/singleton/{all/qemu-iotests-writeback objectstore/bluestore-stupid openstack supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369096/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369096/

    Error reimaging machines: Failed to power on smithi019

[7369149]  rbd/singleton/{all/rbdmap_RBDMAPFILE objectstore/bluestore-comp-zstd openstack supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369149/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369149/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8dcf8e7becbd4ab3acf1997e8b2c8bb4

    SSH connection to smithi159 was lost: 'sudo yum install -y kernel'

[7369152]  rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} thrashers/cache thrashosds-health workloads/rbd_fsx_deep_copy}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369152/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369152/

    Error reimaging machines: 'wait for sentinel on smithi066' reached maximum
    tries (35) after waiting for 1800 seconds

[7369153]  rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369153/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369153/

    Error reimaging machines: Expected smithi098's OS to be rhel 8.4 but found
    centos 8

[7369154]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369154/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369154/

    Error reimaging machines: Failed to power on smithi150

[7369161]  rbd/encryption/{cache/writeback clusters/{fixed-3 openstack} features/readbalance msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks1}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369161/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-15_17:00:18-rbd-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7369161/

    Error reimaging machines: Failed to power on smithi026