2024-04-08T07:25:14.109 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-08T07:25:14.109 INFO:teuthology.results:subset: '399/1024'
2024-04-08T07:25:14.109 INFO:teuthology.results:seed: '3309'
2024-04-08T07:27:33.498 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-08T07:27:33.502 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-08T08:38:39.122 INFO:teuthology.results:Tests finished! gathering results...
2024-04-08T08:38:39.400 INFO:scrape:Found 5 jobs
2024-04-08T08:38:41.958 INFO:scrape:Found 2 distinct failure reasons
2024-04-08T08:38:41.958 INFO:scrape:Failure: "1712563334.9493663 mon.smithi097 (mon.0) 963 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
2024-04-08T08:38:41.958 INFO:scrape:2 jobs: ['7645431', '7645429']
2024-04-08T08:38:41.958 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}', 'tasks/{0-from/quincy']
2024-04-08T08:38:41.958 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}', 'tasks/{0-from/quincy']
2024-04-08T08:38:41.959 INFO:scrape:
2024-04-08T08:38:41.959 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-08T08:38:41.959 INFO:scrape:3 jobs: ['7645432', '7645428', '7645430']
2024-04-08T08:38:41.959 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}']
2024-04-08T08:38:41.959 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'singlethreaded-fuse', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}']
2024-04-08T08:38:41.959 INFO:scrape:
2024-04-08T08:38:41.960 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7645428]  fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:02:08
info:   https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645428/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645428/

    reached maximum tries (51) after waiting for 300 seconds

[7645429]  fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:38
info:   https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645429/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645429/

    "1712563186.5142102 mon.smithi086 (mon.0) 924 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7645430]  fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:46:49
info:   https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645430/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645430/

    reached maximum tries (51) after waiting for 300 seconds

[7645431]  fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:42
info:   https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645431/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645431/

    "1712563334.9493663 mon.smithi097 (mon.0) 963 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7645432]  fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health singlethreaded-fuse syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:47:19
info:   https://pulpito.ceph.com/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645432/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-08_07:24:24-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7645432/

    reached maximum tries (51) after waiting for 300 seconds