2024-02-27T09:21:47.511 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-27T09:21:47.512 INFO:teuthology.results:seed: '7360'
2024-02-27T18:27:04.300 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-27T18:27:04.324 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-27T20:35:14.873 INFO:teuthology.results:Tests finished! gathering results...
2024-02-27T20:35:22.173 INFO:scrape:Found 86 jobs
2024-02-27T20:35:30.215 INFO:scrape:Found 1 distinct failure reasons
2024-02-27T20:35:30.215 INFO:scrape:Failure: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull'
2024-02-27T20:35:30.215 INFO:scrape:65 jobs
2024-02-27T20:35:30.216 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2024-02-27T20:35:30.217 INFO:scrape:suites union: ['1-client', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-upgrade', '3-compat_client/quincy}}', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'tasks/{0-octopus']
2024-02-27T20:35:30.217 INFO:scrape:
2024-02-27T20:35:30.221 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/
failed:  65
dead:    0
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[7574559]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:11
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574559/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574559/
sentry: https://sentry.ceph.com/organizations/ceph/?query=913f2287bf174f0d97fba0701fafe823

    Command failed on smithi151 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574562]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:08
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574562/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574562/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c53a12d018c14e08acdb50bdeb2cb1dc

    Command failed on smithi138 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574563]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:27
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574563/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574563/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c843086ac31a4028b757d1eb2fb3ff9a

    Command failed on smithi066 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574564]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:28
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574564/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574564/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c31be8d67ece4864a71a3d4c5e50fb15

    Command failed on smithi116 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574565]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:20
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574565/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574565/
sentry: https://sentry.ceph.com/organizations/ceph/?query=68917a58afe84f538573fd723c3e5514

    Command failed on smithi161 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574566]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:30
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574566/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574566/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d04d795c8ebc40239bbb77b605d5ab42

    Command failed on smithi043 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574567]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:49
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574567/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574567/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a2979080569a4034b4be30b1dcfde3e4

    Command failed on smithi155 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574568]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:50
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574568/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574568/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20213fe8d18840f39a148b9496d29c80

    Command failed on smithi042 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574569]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:33
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574569/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574569/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b172729e81024cc79f636695cb05fe3f

    Command failed on smithi094 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574572]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:05
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574572/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574572/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a11c9709f8245d8a390d2cfd6b6a723

    Command failed on smithi104 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574573]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:52
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574573/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574573/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca298477c385417cbd34f36c3f6ab45a

    Command failed on smithi173 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574575]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:33
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574575/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574575/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86254ec1251e44f2953541dc471abab7

    Command failed on smithi176 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574577]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:56
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574577/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574577/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4049d2721b834790887cf2c0db96771e

    Command failed on smithi006 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574579]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:39
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574579/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574579/
sentry: https://sentry.ceph.com/organizations/ceph/?query=16734db7efd34fbaa145b9522813bcc6

    Command failed on smithi045 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574580]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:40
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574580/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574580/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec9227eafeab4c06b80db0522c898609

    Command failed on smithi100 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574581]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:56
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574581/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574581/
sentry: https://sentry.ceph.com/organizations/ceph/?query=641dc153e6f74b51a3fdeb245edfac33

    Command failed on smithi027 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574582]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:14
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574582/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574582/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aa1a124c51824b7587bbeee6b8112b49

    Command failed on smithi001 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574583]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:56
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574583/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574583/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80c7fbc2b85e414d9422719c8ec9011b

    Command failed on smithi012 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574584]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:49
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574584/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574584/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5f62857efab4baca675c9c1a6c64366

    Command failed on smithi110 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574586]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:13:18
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574586/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574586/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e3c4c07d41724c7cbd97af4f52cfa072

    Command failed on smithi088 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574587]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:00
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574587/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574587/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b23f449d66144b2aee9bc5dacdfde97

    Command failed on smithi150 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574588]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:25
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574588/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574588/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5779fdabf98b45a9b2d32d1d58a527af

    Command failed on smithi131 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574589]  fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:25:11
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574589/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574589/

    Command failed on smithi005 with status 22: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell 4.36 deep-scrub'

[7574590]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:19
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574590/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574590/
sentry: https://sentry.ceph.com/organizations/ceph/?query=752dc633db12482a9b839e4741da5022

    Command failed on smithi146 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574592]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:57
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574592/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574592/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9bc7d88a4eed4e52a18b4866dd95804f

    Command failed on smithi096 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574593]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:51
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574593/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574593/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e767bce516d4b909f6d27db439bb019

    Command failed on smithi016 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574594]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:59
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574594/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574594/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e456c38be0844a8cb9ba6c6e0639dc1b

    Command failed on smithi018 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574595]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:04
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574595/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574595/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6dd23da82bd44d94bdcace755c268ef7

    Command failed on smithi028 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574596]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:13:47
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574596/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574596/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1bec6c21ceb44d048595408f4a99878e

    Command failed on smithi062 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574597]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:15:20
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574597/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574597/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1633f7c5648b4db4ab2cd78fccb27de4

    Command failed on smithi007 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574598]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:52
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574598/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574598/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01d33c5daa784d7e858b7cc14fb5d7b8

    Command failed on smithi043 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574599]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:12
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574599/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574599/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8466d474aa854c2e804e18f8f725b450

    Command failed on smithi002 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574600]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:22
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574600/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574600/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e263e101e4e84ce08e9f0f0a06bd80df

    Command failed on smithi066 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574603]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:03
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574603/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574603/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5e004dcf86de4ff4b452f4bb1793ecb6

    Command failed on smithi022 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574605]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:12
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574605/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574605/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67a690d954ec4c16ba0e0b8cd228701c

    Command failed on smithi026 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574606]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:42
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574606/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574606/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a679b319263d4fe78fc62cda7a651990

    Command failed on smithi023 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574607]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:51
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574607/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574607/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f5f4632e4fc4803987ed791dee56a0b

    Command failed on smithi017 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574608]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:10
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574608/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4572ab292f4d489dbc89675a3c324e5c

    Command failed on smithi137 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574610]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:24
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574610/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574610/
sentry: https://sentry.ceph.com/organizations/ceph/?query=157007fa804c4e729dfd70ffea5bc84b

    Command failed on smithi123 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574611]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:00
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574611/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574611/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11c535c79a764c668e751ce0b60df58d

    Command failed on smithi003 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574612]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:16
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574612/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574612/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19f5a9edddb94bb9ba3e9796135a3947

    Command failed on smithi177 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574613]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:14:27
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574613/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574613/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c55c123148e4b80adb13cb561bec00b

    Command failed on smithi118 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574615]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:48
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574615/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574615/
sentry: https://sentry.ceph.com/organizations/ceph/?query=319d957c11784fa7b641f9cf8ea32f19

    Command failed on smithi086 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574616]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:13:58
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574616/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574616/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a22f8a84e764446c9b1edcd133c75c85

    Command failed on smithi039 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574618]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:40
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574618/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=225e1ed3e0aa49c79eddfc71ba2ad50c

    Command failed on smithi008 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574619]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:13:27
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574619/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574619/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36e1f53b0a9a4408a38dc2c7e978c5a1

    Command failed on smithi029 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574621]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:47
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574621/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574621/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a7b17f8f38ca4725a6f06c1381f2cc24

    Command failed on smithi055 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574622]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:42
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574622/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574622/
sentry: https://sentry.ceph.com/organizations/ceph/?query=66a2f5e8bbe249cb9594912d7ffa2078

    Command failed on smithi138 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574623]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:17
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574623/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574623/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca226e46b822493a8c5fe5312764986d

    Command failed on smithi047 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574624]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:00
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574624/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574624/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c4ce14d6bb74940ae8e4fae49b6d0b1

    Command failed on smithi116 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574625]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:41
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574625/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574625/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef44edaa45e541a997f813d8a3235290

    Command failed on smithi046 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574626]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:12
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574626/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574626/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ab7df8d36a0452c9cebdb25bc507159

    Command failed on smithi162 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574628]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:50
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574628/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574628/
sentry: https://sentry.ceph.com/organizations/ceph/?query=752bb2b1c8e841878eebd40e66247e0c

    Command failed on smithi132 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574629]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:24
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574629/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574629/
sentry: https://sentry.ceph.com/organizations/ceph/?query=09deb5c39e494397bbcec77a02d6d644

    Command failed on smithi078 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574630]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:34
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574630/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574630/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29f7ba34266d4ea0863ce694c711e46b

    Command failed on smithi080 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574632]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:11:21
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574632/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574632/
sentry: https://sentry.ceph.com/organizations/ceph/?query=590a77ae56804dfbaf3360a438084445

    Command failed on smithi106 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574634]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:59
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574634/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574634/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ded7e068b2ee43288a20fca345a6699c

    Command failed on smithi096 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574635]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:58
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574635/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574635/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d0fbf3802d014440b84aa7fc681f3c7f

    Command failed on smithi043 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574636]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:48
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574636/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574636/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e1c61157458d47eca3fec3e3b84ea93b

    Command failed on smithi066 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574637]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:22
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574637/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574637/
sentry: https://sentry.ceph.com/organizations/ceph/?query=34c5fb597da041179fc150d64586b134

    Command failed on smithi026 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull'

[7574638]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:49
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574638/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574638/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c613bc903d3a481ea84f418f073d6cde

    Command failed on smithi023 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574639]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:09:55
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574639/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574639/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d0f86cb1879f4753944700b0597dba7f

    Command failed on smithi104 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull'

[7574640]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:37
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574640/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574640/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4353f4b12fa949eabf60452156aace97

    Command failed on smithi173 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'

[7574641]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:50
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574641/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574641/
sentry: https://sentry.ceph.com/organizations/ceph/?query=166649f14b4f4bdba988bc70f5bbd7f3

    Command failed on smithi006 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull'

[7574642]  fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:10:18
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574642/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574642/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c877b66e29224a89a73e3af434590d29

    Command failed on smithi138 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    pull'



Pass
=================================================================
[7574558] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:59:48
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574558/

[7574560] fs:upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:30:26
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574560/

[7574561] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}}
time:   00:13:44
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574561/

[7574570] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}}
time:   00:17:33
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574570/

[7574571] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}}
time:   00:20:25
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574571/

[7574574] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:24:07
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574574/

[7574576] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}}
time:   00:25:20
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574576/

[7574578] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}}
time:   00:55:06
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574578/

[7574585] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}}
time:   00:20:50
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574585/

[7574591] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}}
time:   00:24:27
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574591/

[7574601] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}}
time:   01:51:46
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574601/

[7574602] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:29:39
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574602/

[7574604] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}}
time:   00:14:56
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574604/

[7574609] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}}
time:   00:42:55
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574609/

[7574614] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}}
time:   00:19:53
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574614/

[7574617] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:24:36
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574617/

[7574620] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}}
time:   00:51:41
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574620/

[7574627] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}}
time:   00:21:59
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574627/

[7574631] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:23:27
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574631/

[7574633] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}}
time:   00:24:32
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574633/

[7574643] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}}
time:   01:56:36
info:   https://pulpito.ceph.com/vshankar-2024-02-27_09:19:54-fs:upgrade-wip-64502-testing-default-smithi/7574643/