2023-01-25T22:40:37.473 INFO:root:teuthology version: 0.0.1.dev31+g912c45c 2023-01-25T22:40:37.474 INFO:teuthology.results:subset: '111/120000' 2023-01-25T22:40:37.474 INFO:teuthology.results:seed: '5592' 2023-01-26T03:11:16.679 INFO:root:teuthology version: 0.0.1.dev31+g912c45c 2023-01-26T03:11:16.732 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-26T13:22:05.352 INFO:teuthology.results:Tests finished! gathering results... 2023-01-26T13:22:21.243 INFO:scrape:Found 184 jobs 2023-01-26T13:23:12.000 INFO:scrape:Found 13 distinct failure reasons 2023-01-26T13:23:12.001 INFO:scrape:Failure: "2023-01-26T02:24:49.690599+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log 2023-01-26T13:23:12.001 INFO:scrape:115 jobs 2023-01-26T13:23:12.002 INFO:scrape:suites intersection: [] 2023-01-26T13:23:12.003 INFO:scrape:suites union: ['1-ceph', '1-client', '1-upgrade}}', '2-logrotate}', '2-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/iozone}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client-micro', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/nofs/{bluestore-bitmap', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'multimds/yes', 'no-mds-cluster', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides', 'overrides/ignorelist_health', 'overrides/whitelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'session_timeout}', 'supported-random-distros$/{centos_8}', 'tasks/acls}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-readahead}', 'tasks/client-recovery}', 'tasks/client}', 'tasks/damage}', 'tasks/data-scan}', 'tasks/dbench', 'tasks/exports}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/ino_release_cb}', 'tasks/journal-repair}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/mds-flush}', 'tasks/mds-full}', 'tasks/mds_creation_retry}', 'tasks/metrics}', 'tasks/mgr-osd-full}', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/quota}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap-schedule}', 'tasks/snap_schedule_snapdir}', 'tasks/strays}', 'tasks/test_journal_migration}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-octopus', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/basic}}', 'test/misc}}', 'test/snapshot}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}', 'validater/valgrind}'] 2023-01-26T13:23:12.004 INFO:scrape: 2023-01-26T13:23:12.004 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d42d65e724427960623d805c4a5a240507876d19 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-01-26T13:23:12.005 INFO:scrape:1 jobs: ['7138260'] 2023-01-26T13:23:12.005 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-01-26T13:23:12.006 INFO:scrape: 2023-01-26T13:23:12.006 INFO:scrape:Failure: reached maximum tries (150) after waiting for 900 seconds 2023-01-26T13:23:12.006 INFO:scrape:1 jobs: ['7138263'] 2023-01-26T13:23:12.007 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}'] 2023-01-26T13:23:12.007 INFO:scrape: 2023-01-26T13:23:12.007 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi161 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' 2023-01-26T13:23:12.008 INFO:scrape:1 jobs: ['7138294'] 2023-01-26T13:23:12.008 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/dbench}', 'wsync/{yes}}'] 2023-01-26T13:23:12.009 INFO:scrape: 2023-01-26T13:23:12.009 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d42d65e724427960623d805c4a5a240507876d19 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-01-26T13:23:12.009 INFO:scrape:1 jobs: ['7138405'] 2023-01-26T13:23:12.010 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-01-26T13:23:12.010 INFO:scrape: 2023-01-26T13:23:12.011 INFO:scrape:Failure: "2023-01-26T04:12:31.442761+0000 mds.e (mds.0) 1 : cluster [WRN] client.4517 isn't responding to mclientcaps(revoke), ino 0x1000000318b pending pAsLsXsFsx issued pAsLsXsFsxr, sent 303.920557 seconds ago" in cluster log 2023-01-26T13:23:12.011 INFO:scrape:1 jobs: ['7138423'] 2023-01-26T13:23:12.011 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-01-26T13:23:12.012 INFO:scrape: 2023-01-26T13:23:12.013 INFO:scrape:Failure: "2023-01-26T01:20:34.355603+0000 osd.2 (osd.2) 12 : cluster [ERR] 5.1fs0 deep-scrub : stat mismatch, got 0/2 objects, 0/0 clones, 0/2 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/8388608 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2023-01-26T13:23:12.013 INFO:scrape:1 jobs: ['7138329'] 2023-01-26T13:23:12.013 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-01-26T13:23:12.014 INFO:scrape: 2023-01-26T13:23:12.014 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-01-26T13:23:12.014 INFO:scrape:2 jobs: ['7138279', '7138406'] 2023-01-26T13:23:12.015 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-01-26T13:23:12.015 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-01-26T13:23:12.016 INFO:scrape: 2023-01-26T13:23:12.016 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d42d65e724427960623d805c4a5a240507876d19 2023-01-26T13:23:12.016 INFO:scrape:2 jobs: ['7138413', '7138346'] 2023-01-26T13:23:12.017 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-01-26T13:23:12.017 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-01-26T13:23:12.018 INFO:scrape: 2023-01-26T13:23:12.018 INFO:scrape:Crash: "2023-01-26T02:54:59.592994+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log 2023-01-26T13:23:12.018 INFO:scrape:ceph version 17.2.5-703-gd42d65e7 (d42d65e724427960623d805c4a5a240507876d19) quincy (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe46d5e6b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe46e5e8493] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2695ff) [0x7fe46e5e85ff] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x1278) [0x564b06360958] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xd75) [0x564b06068315] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5db) [0x564b0606a7ab] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b0606adec] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564b0605445f] 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe46e862e68] 12: (DispatchQueue::entry()+0x50f) [0x7fe46e8602af] 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe46e927ad1] 14: /lib64/libpthread.so.0(+0x814a) [0x7fe46d5dc14a] 15: clone() 2023-01-26T13:23:12.019 INFO:scrape:2 jobs: ['7138407', '7138253'] 2023-01-26T13:23:12.019 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-01-26T13:23:12.020 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snapshots}'] 2023-01-26T13:23:12.020 INFO:scrape: 2023-01-26T13:23:12.020 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi012.front.sepia.ceph.com: ['type=AVC msg=audit(1674688493.679:195): avc: denied { node_bind } for pid=1390 comm="ping" saddr=172.21.15.12 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2023-01-26T13:23:12.021 INFO:scrape:1 jobs: ['7138259'] 2023-01-26T13:23:12.021 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/fs/misc}', 'wsync/{no}}'] 2023-01-26T13:23:12.022 INFO:scrape: 2023-01-26T13:23:12.022 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-01-26T13:23:12.022 INFO:scrape:2 jobs: ['7138399', '7138302'] 2023-01-26T13:23:12.023 INFO:scrape:suites intersection: [] 2023-01-26T13:23:12.023 INFO:scrape:suites union: ['1-mount', '1-rename', '2-workload/dbench', 'distro/single-container-host', 'distro/single-container-host}', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'overrides/ignorelist_health}'] 2023-01-26T13:23:12.023 INFO:scrape: 2023-01-26T13:23:12.024 INFO:scrape:Dead: b'2023-01-26T13:16:21.825 DEBUG:teuthology.exit:Finished running handlers' 2023-01-26T13:23:12.024 INFO:scrape:1 jobs: ['7138348'] 2023-01-26T13:23:12.025 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-01-26T13:23:12.025 INFO:scrape: 2023-01-26T13:23:12.033 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/ failed: 130 dead: 1 running: 0 waiting: 0 queued: 0 passed: 53 Fail ================================================================= [7138248] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:21:42 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138248/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138248/ "2023-01-25T23:02:23.504924+0000 mgr.x (mgr.4115) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138249] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:35:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138249/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138249/ "2023-01-25T23:06:52.893487+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138250] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:22:20 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138250/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138250/ "2023-01-25T23:06:32.927418+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138252] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:18:34 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138252/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138252/ "2023-01-25T23:11:44.057258+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138253] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:28:41 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138253/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138253/ "2023-01-25T23:21:51.462591+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138255] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:28:22 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138255/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138255/ "2023-01-25T23:20:28.258567+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138256] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:22:38 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138256/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138256/ "2023-01-25T23:24:41.376863+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138257] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:25:43 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138257/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138257/ "2023-01-25T23:27:54.090474+0000 mgr.y (mgr.14103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138258] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:16:48 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138258/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138258/ "2023-01-25T23:19:21.750690+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138259] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 00:53:38 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138259/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138259/ SELinux denials found on ubuntu@smithi012.front.sepia.ceph.com: ['type=AVC msg=audit(1674688493.679:195): avc: denied { node_bind } for pid=1390 comm="ping" saddr=172.21.15.12 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7138260] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:23:45 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138260/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138260/ sentry: https://sentry.ceph.com/organizations/ceph/?query=346bd55386cc419ba2c12bbf3587c641 Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d42d65e724427960623d805c4a5a240507876d19 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7138261] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:21 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138261/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138261/ "2023-01-25T23:24:38.362806+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138263] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:41:17 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138263/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138263/ reached maximum tries (150) after waiting for 900 seconds [7138264] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138264/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138264/ "2023-01-25T23:34:13.470243+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138266] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138266/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138266/ "2023-01-25T23:30:59.514674+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138267] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:14:23 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138267/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138267/ "2023-01-25T23:31:50.816303+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138269] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:12:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138269/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138269/ "2023-01-25T23:36:05.048638+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138271] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 01:33:59 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138271/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138271/ "2023-01-25T23:36:49.470877+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138273] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:27:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138273/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138273/ "2023-01-25T23:50:06.026129+0000 mgr.y (mgr.14098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138274] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:21:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138274/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138274/ "2023-01-25T23:50:12.585701+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138275] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138275/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138275/ "2023-01-25T23:51:31.745197+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138278] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:11:04 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138278/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138278/ "2023-01-25T23:59:32.775047+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138279] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:01:08 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138279/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e08eb4fa5c1498f99912358a675c2ce Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7138280] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:19:24 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138280/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138280/ "2023-01-25T23:53:35.310216+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138282] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:51:15 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138282/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138282/ "2023-01-25T23:56:10.070723+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138283] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:24:43 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138283/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138283/ "2023-01-25T23:53:30.720427+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138284] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:16:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138284/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138284/ "2023-01-25T23:53:47.869972+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138285] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:15:34 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138285/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138285/ "2023-01-25T23:56:11.711020+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138287] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:12:05 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138287/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138287/ "2023-01-25T23:55:26.876423+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138288] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:21:06 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138288/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138288/ "2023-01-26T00:01:50.136047+0000 mgr.x (mgr.14106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138292] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:12:22 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138292/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138292/ "2023-01-26T00:02:48.877358+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138294] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 01:50:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138294/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138294/ error during scrub thrashing: Command failed on smithi161 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7138295] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138295/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138295/ "2023-01-26T00:08:52.854878+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138296] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:28:24 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138296/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138296/ "2023-01-26T00:21:33.138975+0000 mgr.y (mgr.14100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138297] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:23:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138297/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138297/ "2023-01-26T00:13:54.847951+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138298] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:18:43 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138298/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138298/ "2023-01-26T00:23:27.966113+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138299] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 01:50:26 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138299/ "2023-01-26T00:24:05.631472+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138300] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:15:09 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138300/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138300/ "2023-01-26T00:21:07.120578+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138302] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:22:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138302/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eabb8494a7194314842604cce3dcdb42 timeout expired in wait_until_healthy [7138303] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138303/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138303/ "2023-01-26T00:28:36.258998+0000 mon.a (mon.0) 322 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7138305] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138305/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138305/ "2023-01-26T00:24:59.477543+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138308] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:41:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138308/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138308/ "2023-01-26T00:23:40.899811+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138311] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138311/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138311/ "2023-01-26T00:33:30.661439+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138313] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:52:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138313/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138313/ "2023-01-26T00:38:41.517394+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138314] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} ----------------------------------------------------------------- time: 00:10:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138314/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138314/ "2023-01-26T00:39:20.084411+0000 mgr.x (mgr.4507) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138316] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:26:55 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138316/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138316/ "2023-01-26T00:47:49.341402+0000 mgr.y (mgr.14108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138317] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:06:13 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138317/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138317/ "2023-01-26T00:55:29.180094+0000 mgr.x (mgr.4631) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138318] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:19:59 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138318/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138318/ "2023-01-26T00:47:05.346723+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138320] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} ----------------------------------------------------------------- time: 00:11:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138320/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138320/ "2023-01-26T00:42:51.434212+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138321] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:52:06 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138321/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138321/ "2023-01-26T00:56:58.760480+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138322] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:12:27 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138322/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138322/ "2023-01-26T00:43:51.830671+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138324] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:37:41 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138324/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138324/ "2023-01-26T00:58:28.060048+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138325] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:19:00 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138325/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138325/ "2023-01-26T00:51:01.990621+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138326/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138326/ "2023-01-26T00:51:55.653262+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138327] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138327/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138327/ "2023-01-26T00:55:47.817741+0000 mgr.x (mgr.24100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138329] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:32:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138329/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138329/ "2023-01-26T01:20:34.355603+0000 osd.2 (osd.2) 12 : cluster [ERR] 5.1fs0 deep-scrub : stat mismatch, got 0/2 objects, 0/0 clones, 0/2 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/8388608 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log [7138330] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:26:25 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138330/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138330/ "2023-01-26T00:55:10.046221+0000 mgr.y (mgr.4112) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138332] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:54:29 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138332/ "2023-01-26T01:10:48.414250+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138333] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138333/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138333/ "2023-01-26T01:03:39.832347+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138334] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138334/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138334/ "2023-01-26T01:04:02.489692+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138335] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:13 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138335/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138335/ "2023-01-26T01:09:24.953142+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138337] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138337/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138337/ "2023-01-26T01:14:34.641177+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138338] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:20:17 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138338/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138338/ "2023-01-26T01:20:22.860382+0000 mgr.x (mgr.14100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138339] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138339/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138339/ "2023-01-26T01:19:14.487264+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138340] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:18:40 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138340/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138340/ "2023-01-26T01:15:41.207724+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138342] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:51:29 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138342/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138342/ "2023-01-26T01:19:24.803587+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138343] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:55:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138343/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138343/ "2023-01-26T01:20:51.269288+0000 mgr.x (mgr.4112) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138344] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:29:41 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138344/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138344/ "2023-01-26T01:21:58.132557+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138345] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:19:21 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138345/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138345/ "2023-01-26T01:33:03.337943+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138346] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138346/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1eaa79588d76477ebea9a4f8529b690c Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d42d65e724427960623d805c4a5a240507876d19 [7138347] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 01:05:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138347/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138347/ "2023-01-26T01:27:50.511204+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138349] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:25:13 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138349/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138349/ "2023-01-26T01:35:51.787453+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:11:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138351/ "2023-01-26T01:32:14.181076+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138352] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:40:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138352/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138352/ "2023-01-26T01:29:25.256269+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138353] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:23:29 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138353/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138353/ "2023-01-26T01:42:26.787013+0000 mgr.x (mgr.14151) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138354] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:22:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138354/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138354/ "2023-01-26T01:41:53.794466+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138355] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:21:22 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138355/ "2023-01-26T01:36:22.012919+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138356] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:36:24 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138356/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138356/ "2023-01-26T01:41:34.819951+0000 mgr.x (mgr.4116) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138357] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138357/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138357/ "2023-01-26T01:38:38.719156+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138358] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:12:07 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138358/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138358/ "2023-01-26T01:40:50.605791+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138359] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:14:48 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138359/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138359/ "2023-01-26T01:41:04.732534+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138360] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:17:28 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138360/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138360/ "2023-01-26T01:50:03.641032+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138364] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:29:15 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138364/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138364/ "2023-01-26T01:50:19.690345+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138365] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:28:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138365/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138365/ "2023-01-26T02:00:42.278014+0000 mgr.y (mgr.34099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138366] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:15:31 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138366/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138366/ "2023-01-26T01:52:14.475246+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138367] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:17:58 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138367/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138367/ "2023-01-26T01:53:37.495032+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138368] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138368/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138368/ "2023-01-26T02:03:50.413155+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138372] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:46:33 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138372/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138372/ "2023-01-26T02:10:38.348652+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138374] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:46:33 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138374/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138374/ "2023-01-26T02:04:00.575139+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:12:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138376/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138376/ "2023-01-26T02:06:01.986121+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138377] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 01:31:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138377/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138377/ "2023-01-26T02:15:17.796722+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138378] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:00 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138378/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138378/ "2023-01-26T02:11:45.358719+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138379] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 01:08:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138379/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138379/ "2023-01-26T02:10:08.875064+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138380] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138380/ "2023-01-26T02:11:54.746018+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138382] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:20:23 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138382/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138382/ "2023-01-26T02:20:27.514724+0000 mgr.x (mgr.14098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138383] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:12:01 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138383/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138383/ "2023-01-26T02:15:45.835946+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138384] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:12:27 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138384/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138384/ "2023-01-26T02:16:01.430957+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138385] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:17:12 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138385/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138385/ "2023-01-26T02:24:49.690599+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138388] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:51:47 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138388/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138388/ "2023-01-26T02:35:42.039821+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138389] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:20:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138389/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138389/ "2023-01-26T02:28:06.042963+0000 mgr.y (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138391] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:26:12 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138391/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138391/ "2023-01-26T02:34:26.692724+0000 mgr.y (mgr.14107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138392] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:14:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138392/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138392/ "2023-01-26T02:28:40.076678+0000 mgr.x (mgr.4115) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138393] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:24:51 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138393/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138393/ "2023-01-26T02:37:23.433599+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138395] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:55:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138395/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138395/ "2023-01-26T02:38:25.910165+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138397] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138397/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138397/ "2023-01-26T02:32:54.244166+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138398] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138398/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138398/ "2023-01-26T02:34:44.802558+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138399] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:22:38 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138399/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=010487a302b8459b9e143f6b50c3f8f0 timeout expired in wait_until_healthy [7138400] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:11:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138400/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138400/ "2023-01-26T02:40:50.245959+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138403] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:57:17 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138403/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138403/ "2023-01-26T02:45:09.036903+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138404] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138404/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138404/ "2023-01-26T02:49:04.824276+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138405] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:26:22 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138405/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab44d332291647028caad95cd97fb8fe Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d42d65e724427960623d805c4a5a240507876d19 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7138406] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:45:06 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138406/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dea6dc8fb1cf48a78aeb7c2ac031720e Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7138407] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:21:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138407/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138407/ "2023-01-26T02:54:59.592994+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138408] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:27:24 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138408/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138408/ "2023-01-26T03:05:06.950867+0000 mgr.x (mgr.14098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138409] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138409/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138409/ "2023-01-26T02:59:09.447331+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138411] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:39:01 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138411/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138411/ "2023-01-26T03:09:26.060941+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138412] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138412/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138412/ "2023-01-26T03:10:34.630224+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138413] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138413/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f2d694660624fd49c218d0737957307 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d42d65e724427960623d805c4a5a240507876d19 [7138414] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138414/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138414/ "2023-01-26T03:01:49.427876+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138415] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} ----------------------------------------------------------------- time: 00:11:44 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138415/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138415/ "2023-01-26T03:05:28.928915+0000 mgr.x (mgr.14103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138416] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:17:48 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138416/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138416/ "2023-01-26T03:02:17.102972+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138417] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:09:02 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138417/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138417/ "2023-01-26T03:02:54.992526+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138418] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:14:36 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138418/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138418/ "2023-01-26T03:04:26.902793+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138421] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:18:33 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138421/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138421/ "2023-01-26T03:17:25.667267+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138422] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:35:08 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138422/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138422/ "2023-01-26T03:13:52.557775+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138423] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:41:36 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138423/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138423/ "2023-01-26T04:12:31.442761+0000 mds.e (mds.0) 1 : cluster [WRN] client.4517 isn't responding to mclientcaps(revoke), ino 0x1000000318b pending pAsLsXsFsx issued pAsLsXsFsxr, sent 303.920557 seconds ago" in cluster log [7138424] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:17:04 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138424/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138424/ "2023-01-26T03:19:52.520252+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138425] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138425/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138425/ "2023-01-26T03:23:19.565993+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138427] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138427/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138427/ "2023-01-26T03:22:21.149469+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7138429] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:12:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138429/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138429/ "2023-01-26T03:24:15.405442+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log Dead ================================================================= [7138348] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138348/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138348/ hit max job timeout Pass ================================================================= [7138247] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:44 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138247/ [7138251] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:24:55 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138251/ [7138254] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138254/ [7138262] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:11:10 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138262/ [7138265] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:58 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138265/ [7138268] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:12:38 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138268/ [7138270] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 02:16:42 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138270/ [7138272] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:15:30 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138272/ [7138276] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:13:55 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138276/ [7138277] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:53:28 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138277/ [7138281] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:32 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138281/ [7138286] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:40:54 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138286/ [7138289] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:28:44 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138289/ [7138290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:26:38 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138290/ [7138291] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:42 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138291/ [7138293] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:19:50 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138293/ [7138301] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:12:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138301/ [7138304] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:19:29 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138304/ [7138306] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} time: 01:56:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138306/ [7138307] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138307/ [7138309] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:48:07 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138309/ [7138310] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:16 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138310/ [7138312] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:35:18 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138312/ [7138315] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:19:46 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138315/ [7138319] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:03:05 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138319/ [7138323] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:52 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138323/ [7138328] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:29:36 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138328/ [7138331] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:12:35 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138331/ [7138336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:31:07 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138336/ [7138341] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:22 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138341/ [7138350] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} time: 00:37:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138350/ [7138361] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:23 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138361/ [7138362] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:24:29 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138362/ [7138363] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:39:27 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138363/ [7138369] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:06 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138369/ [7138370] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:01 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138370/ [7138371] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:50:14 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138371/ [7138373] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:17 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138373/ [7138375] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:38:42 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138375/ [7138381] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:58:56 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138381/ [7138386] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:38:27 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138386/ [7138387] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:36:16 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138387/ [7138390] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:41 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138390/ [7138394] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:26:28 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138394/ [7138396] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:08 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138396/ [7138401] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:13:19 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138401/ [7138402] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:47:34 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138402/ [7138410] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138410/ [7138419] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138419/ [7138420] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:28:21 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138420/ [7138426] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:05 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138426/ [7138428] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:24:36 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138428/ [7138430] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:37 info: http://pulpito.ceph.com/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138430/