2024-04-10T01:32:01.006 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-10T01:32:01.006 INFO:teuthology.results:subset: '111/120000'
2024-04-10T01:32:01.006 INFO:teuthology.results:seed: '7233'
2024-04-10T02:48:44.222 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-10T02:48:44.247 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-10T06:05:00.528 INFO:teuthology.results:Tests finished! gathering results...
2024-04-10T06:05:06.876 INFO:scrape:Found 70 jobs
2024-04-10T06:05:50.745 INFO:scrape:Found 9 distinct failure reasons
2024-04-10T06:05:50.746 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-10T06:05:50.746 INFO:scrape:5 jobs: ['7648882', '7648857', '7648904', '7648930', '7648895']
2024-04-10T06:05:50.746 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-10T06:05:50.746 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-10T06:05:50.747 INFO:scrape:
2024-04-10T06:05:50.747 INFO:scrape:Failure: timeout expired in wait_until_healthy
2024-04-10T06:05:50.747 INFO:scrape:15 jobs: ['7648878', '7648896', '7648888', '7648875', '7648854', '7648858', '7648899', '7648864', '7648892', '7648922', '7648861', '7648928', '7648914', '7648881', '7648943']
2024-04-10T06:05:50.747 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
2024-04-10T06:05:50.748 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04']
2024-04-10T06:05:50.748 INFO:scrape:
2024-04-10T06:05:50.748 INFO:scrape:Failure: "2024-04-10T02:50:00.000137+0000 mon.a (mon.0) 2518 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log
2024-04-10T06:05:50.748 INFO:scrape:17 jobs: ['7648868', '7648926', '7648938', '7648903', '7648880', '7648856', '7648860', '7648863', '7648894', '7648916', '7648910', '7648883', '7648886', '7648890', '7648872', '7648876', '7648932']
2024-04-10T06:05:50.748 INFO:scrape:suites intersection: []
2024-04-10T06:05:50.749 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'workload/{ec-rados-default']
2024-04-10T06:05:50.749 INFO:scrape:
2024-04-10T06:05:50.749 INFO:scrape:Failure: Command failed on smithi087 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fad5473c-f6e2-11ee-b64a-cb9ed24678a4 -e sha1=3a866975a60f6dc859246f6f19fd8ddb4db2cc2e -- bash -c 'ceph orch ps'"
2024-04-10T06:05:50.749 INFO:scrape:3 jobs: ['7648918', '7648940', '7648870']
2024-04-10T06:05:50.749 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-10T06:05:50.750 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-10T06:05:50.750 INFO:scrape:
2024-04-10T06:05:50.750 INFO:scrape:Failure: "1712715171.255007 mon.a (mon.0) 524 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2024-04-10T06:05:50.750 INFO:scrape:5 jobs: ['7648884', '7648908', '7648865', '7648871', '7648934']
2024-04-10T06:05:50.750 INFO:scrape:suites intersection: ['0-start']
2024-04-10T06:05:50.750 INFO:scrape:suites union: ['0-start', '1-tasks', '1-tasks}', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc}', 'workload/{ec-rados-default']
2024-04-10T06:05:50.750 INFO:scrape:
2024-04-10T06:05:50.750 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'
2024-04-10T06:05:50.751 INFO:scrape:1 jobs: ['7648898']
2024-04-10T06:05:50.751 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream']
2024-04-10T06:05:50.751 INFO:scrape:
2024-04-10T06:05:50.751 INFO:scrape:Failure: Command failed on smithi136 with status 1: "sudo egrep 'FileStore has been deprecated and is no longer supported' /var/log/ceph/ceph-osd.1.log"
2024-04-10T06:05:50.751 INFO:scrape:1 jobs: ['7648902']
2024-04-10T06:05:50.751 INFO:scrape:suites: ['-', '1-ceph-install/quincy', '2', 'objectstore/filestore-xfs', 'start}', 'ubuntu_latest}', 'upgrade', 'upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack']
2024-04-10T06:05:50.751 INFO:scrape:
2024-04-10T06:05:50.752 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific
2024-04-10T06:05:50.752 INFO:scrape:1 jobs: ['7648900']
2024-04-10T06:05:50.752 INFO:scrape:suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream}']
2024-04-10T06:05:50.752 INFO:scrape:
2024-04-10T06:05:50.752 INFO:scrape:Failure: Command failed on smithi190 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'
2024-04-10T06:05:50.752 INFO:scrape:1 jobs: ['7648920']
2024-04-10T06:05:50.752 INFO:scrape:suites: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/dbench}}', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-install', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap']
2024-04-10T06:05:50.752 INFO:scrape:
2024-04-10T06:05:50.758 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/
failed:  49
dead:    0
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[7648854]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:33:07
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648854/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648854/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02204b8bd04a4e8c9f43818fad3ee51a

    timeout expired in wait_until_healthy

[7648856]  upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:59:40
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648856/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648856/

    "2024-04-10T02:30:00.000153+0000 mon.a (mon.0) 1522 : cluster 3 [WRN]
    OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

[7648857]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:52:49
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648857/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648857/

    reached maximum tries (51) after waiting for 300 seconds

[7648858]  upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:51:48
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648858/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648858/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79bfdc9ef70046fdbe96541b71259741

    timeout expired in wait_until_healthy

[7648860]  upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   03:31:11
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648860/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648860/

    "2024-04-10T02:13:37.471439+0000 mon.a (mon.0) 16 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in
    cluster log

[7648861]  upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:57:27
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648861/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648861/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d9767014cee24eeaad265330c35d8904

    timeout expired in wait_until_healthy

[7648863]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:38:48
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648863/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648863/

    "2024-04-10T02:06:01.098790+0000 mon.a (mon.0) 197 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648864]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:32:08
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648864/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648864/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7bdff91a8a424fd191b9f6261d428660

    timeout expired in wait_until_healthy

[7648865]  upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks}
-----------------------------------------------------------------
time:   00:30:36
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648865/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648865/

    "1712714368.4796205 mon.a (mon.0) 511 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7648868]  upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:48:00
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648868/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648868/

    "2024-04-10T02:50:00.000137+0000 mon.a (mon.0) 2518 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in
    cluster log

[7648870]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:31:15
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648870/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648870/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cbdf64abbe284115957aaa930d683f48

    Command failed on smithi016 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a95001e0-f6dd-11ee-b64a-cb9ed24678a4 -e
    sha1=3a866975a60f6dc859246f6f19fd8ddb4db2cc2e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7648871]  upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:26:18
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648871/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648871/

    "1712714496.0207624 mon.a (mon.0) 525 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7648872]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   03:15:48
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648872/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648872/

    "2024-04-10T02:20:00.000157+0000 mon.a (mon.0) 681 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

[7648875]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:32:38
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648875/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648875/
sentry: https://sentry.ceph.com/organizations/ceph/?query=88e03e5cbb71404bb2a119414506878d

    timeout expired in wait_until_healthy

[7648876]  upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   03:30:02
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648876/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648876/

    "2024-04-10T02:09:13.850946+0000 mon.a (mon.0) 179 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648878]  upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:51:20
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648878/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648878/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f2f720d189e469b8bbfd746785084e6

    timeout expired in wait_until_healthy

[7648880]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   03:11:13
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648880/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648880/

    "2024-04-10T02:30:00.000144+0000 mon.a (mon.0) 754 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

[7648881]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:33:34
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648881/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648881/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0eb86c551a7a4b6dad3f412eb5ebacfb

    timeout expired in wait_until_healthy

[7648882]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:33
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648882/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648882/

    reached maximum tries (51) after waiting for 300 seconds

[7648883]  upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:48:53
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648883/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648883/

    "2024-04-10T03:10:00.000094+0000 mon.a (mon.0) 3230 : cluster 3 [WRN]
    TELEMETRY_CHANGED: Telemetry requires re-opt-in" in cluster log

[7648884]  upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:28:26
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648884/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648884/

    "1712715171.255007 mon.a (mon.0) 524 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7648886]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:34:57
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648886/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648886/

    "2024-04-10T02:23:27.734273+0000 mon.a (mon.0) 320 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648888]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:40:34
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648888/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648888/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bcbc24e203f447cab0dd7992ccb16b04

    timeout expired in wait_until_healthy

[7648890]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   03:20:30
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648890/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648890/

    "2024-04-10T04:00:00.000126+0000 mon.a (mon.0) 7127 : cluster 4 [ERR]
    OSD_SCRUB_ERRORS: 1 scrub errors" in cluster log

[7648892]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:45:18
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648892/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ca9ebab01934eb3b416ee4dfb3cae1f

    timeout expired in wait_until_healthy

[7648894]  upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:45:00
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648894/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648894/

    "2024-04-10T02:21:19.294443+0000 mon.a (mon.0) 179 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648895]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:07
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648895/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648895/

    reached maximum tries (51) after waiting for 300 seconds

[7648896]  upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:45:32
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648896/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e39ca73b4284a97927a4e059fcf5f56

    timeout expired in wait_until_healthy

[7648898]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   02:04:06
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648898/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648898/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6de3bdcd775b4380ad88c06a1951f0ec

    Command failed (workunit test rbd/test_librbd.sh) on smithi154 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'

[7648899]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:33:25
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648899/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648899/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c949101dd325430294517a73db2cd119

    timeout expired in wait_until_healthy

[7648900]  upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks}
-----------------------------------------------------------------
time:   00:04:45
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648900/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648900/
sentry: https://sentry.ceph.com/organizations/ceph/?query=12ac9c13b3c64f678a9e36e6b42ee151

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci
    fic

[7648902]  upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:09:59
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648902/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=34e0fe85ac7745938944a35af7d44b94

    Command failed on smithi136 with status 1: "sudo egrep 'FileStore has been
    deprecated and is no longer supported' /var/log/ceph/ceph-osd.1.log"

[7648903]  upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:47:04
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648903/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648903/

    "2024-04-10T03:20:00.000128+0000 mon.a (mon.0) 3092 : cluster 3 [WRN]
    TELEMETRY_CHANGED: Telemetry requires re-opt-in" in cluster log

[7648904]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:35
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648904/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648904/

    reached maximum tries (51) after waiting for 300 seconds

[7648908]  upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:28:18
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648908/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648908/

    "1712715811.530643 mon.a (mon.0) 509 : cluster [WRN] Health check failed:
    Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded
    (PG_DEGRADED)" in cluster log

[7648910]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   03:15:41
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648910/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648910/

    "2024-04-10T02:42:15.379509+0000 mon.a (mon.0) 16 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in
    cluster log

[7648914]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:40:45
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648914/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f624e3f46d8b43e68ce96a0543018c62

    timeout expired in wait_until_healthy

[7648916]  upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   03:14:21
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648916/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648916/

    "2024-04-10T02:40:50.521599+0000 mon.a (mon.0) 179 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648918]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:17:24
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648918/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e2c1fd00e85e442f90905eaadaf1fbc0

    Command failed on smithi087 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fad5473c-f6e2-11ee-b64a-cb9ed24678a4 -e
    sha1=3a866975a60f6dc859246f6f19fd8ddb4db2cc2e -- bash -c 'ceph orch ps'"

[7648920]  upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}}
-----------------------------------------------------------------
time:   00:11:24
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648920/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c96b84b1662450fb3de0e8697815a62

    Command failed on smithi190 with status 1: 'sudo yum -y upgrade ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd'

[7648922]  upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:43:59
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648922/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a54bfa77e48497eb41b7442d85252b8

    timeout expired in wait_until_healthy

[7648926]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:47:52
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648926/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648926/

    "2024-04-10T04:30:00.000142+0000 mon.a (mon.0) 7717 : cluster 4 [ERR]
    OSD_SCRUB_ERRORS: 4 scrub errors" in cluster log

[7648928]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   00:40:42
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648928/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc7e84244a0f4ceaaee8b1a26191a613

    timeout expired in wait_until_healthy

[7648930]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:15
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648930/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648930/

    reached maximum tries (51) after waiting for 300 seconds

[7648932]  upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:58:09
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648932/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648932/

    "2024-04-10T02:51:55.008997+0000 mon.a (mon.0) 178 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7648934]  upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   01:40:53
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648934/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648934/

    "1712718074.8551445 mon.a (mon.0) 509 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7648938]  upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   03:19:51
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648938/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648938/

    "2024-04-10T04:30:00.000258+0000 mon.a (mon.0) 5883 : cluster 4 [ERR]
    OSD_SCRUB_ERRORS: 55 scrub errors" in cluster log

[7648940]  upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:17:10
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648940/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648940/
sentry: https://sentry.ceph.com/organizations/ceph/?query=08b3c0ba83c64564b7d9755bd2d091da

    Command failed on smithi063 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ace850f2-f6e5-11ee-b64a-cb9ed24678a4 -e
    sha1=3a866975a60f6dc859246f6f19fd8ddb4db2cc2e -- bash -c 'ceph orch ps'"

[7648943]  upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:45:20
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648943/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648943/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e54eb1681bc5438cb0db0f66ee2cca56

    timeout expired in wait_until_healthy



Pass
=================================================================
[7648855] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:21
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648855/

[7648859] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:20:08
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648859/

[7648862] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:14
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648862/

[7648866] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:26:11
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648866/

[7648867] upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest}
time:   00:09:58
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648867/

[7648869] upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:11:55
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648869/

[7648873] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}}
time:   00:38:35
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648873/

[7648874] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:50
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648874/

[7648877] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:18
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648877/

[7648879] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:21:27
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648879/

[7648885] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:25:21
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648885/

[7648887] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}}
time:   00:16:35
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648887/

[7648889] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:35:43
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648889/

[7648891] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:38
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648891/

[7648893] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}}
time:   00:15:46
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648893/

[7648897] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:20:11
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648897/

[7648901] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:25:32
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648901/

[7648906] upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:12:08
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648906/

[7648912] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:07
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648912/

[7648924] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:21:50
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648924/

[7648936] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:23:54
info:   https://pulpito.ceph.com/yuriw-2024-04-09_14:58:21-upgrade-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648936/