2024-03-08T12:14:16.182 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-08T12:14:16.183 INFO:teuthology.results:subset: '1/8' 2024-03-08T12:14:16.183 INFO:teuthology.results:seed: '2995' 2024-03-08T13:49:54.999 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-08T13:49:55.004 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-09T01:12:52.659 INFO:teuthology.results:Tests finished! gathering results... 2024-03-09T01:12:59.870 INFO:scrape:Found 99 jobs 2024-03-09T01:13:02.446 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:02.447 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:02.447 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:05.787 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:05.787 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:05.788 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:07.910 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:07.910 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:07.910 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:09.270 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:09.270 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:09.270 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:11.211 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:11.211 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:11.211 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:17.192 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:17.192 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:17.193 WARNING:scrape:Saw end of BT but not start 2024-03-09T01:13:20.062 INFO:scrape:Found 7 distinct failure reasons 2024-03-09T01:13:20.062 INFO:scrape:Failure: "2024-03-08T13:34:06.870001+0000 mon.smithi072 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log 2024-03-09T01:13:20.063 INFO:scrape:52 jobs 2024-03-09T01:13:20.063 INFO:scrape:suites intersection: [] 2024-03-09T01:13:20.063 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'task/test_orch_cli_mon}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.064 INFO:scrape: 2024-03-09T01:13:20.064 INFO:scrape:Failure: "2024-03-08T13:20:00.000230+0000 mon.a (mon.0) 1628 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log 2024-03-09T01:13:20.064 INFO:scrape:1 jobs: ['7585594'] 2024-03-09T01:13:20.064 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/radosbench', 'fixed-2', 'msgr/async-v1only', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'root}'] 2024-03-09T01:13:20.064 INFO:scrape: 2024-03-09T01:13:20.064 INFO:scrape:Dead: b'2024-03-09T00:26:32.916 DEBUG:teuthology.exit:Finished running handlers' 2024-03-09T01:13:20.064 INFO:scrape:3 jobs: ['7585651', '7585656', '7585663'] 2024-03-09T01:13:20.064 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start'] 2024-03-09T01:13:20.064 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', '2-services/nfs-ingress', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc'] 2024-03-09T01:13:20.065 INFO:scrape: 2024-03-09T01:13:20.065 INFO:scrape:Failure: "1709907842.0050786 mon.smithi135 (mon.0) 718 : cluster [WRN] Replacing daemon mds.cephfs.smithi135.pfyhhg as rank 0 with standby daemon mds.cephfs.smithi144.ccbdwu" in cluster log 2024-03-09T01:13:20.065 INFO:scrape:2 jobs: ['7585655', '7585621'] 2024-03-09T01:13:20.065 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.065 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.065 INFO:scrape: 2024-03-09T01:13:20.065 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27f99b14d082e0d773119df169ad98479243bcba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-03-09T01:13:20.065 INFO:scrape:1 jobs: ['7585637'] 2024-03-09T01:13:20.065 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-03-09T01:13:20.065 INFO:scrape: 2024-03-09T01:13:20.066 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-09T01:13:20.066 INFO:scrape:2 jobs: ['7585671', '7585638'] 2024-03-09T01:13:20.066 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.066 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.066 INFO:scrape: 2024-03-09T01:13:20.066 INFO:scrape:Failure: Failed to reconnect to smithi158 2024-03-09T01:13:20.066 INFO:scrape:1 jobs: ['7585607'] 2024-03-09T01:13:20.066 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-09T01:13:20.067 INFO:scrape: 2024-03-09T01:13:20.070 INFO:teuthology.results:Sending results to gabrioux@ibm.com: Test Run: gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/ failed: 59 dead: 3 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7585586] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:58:04 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585586/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585586/ "1709903525.8367467 mon.smithi077 (mon.0) 696 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7585587] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:35:07 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585587/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585587/ "2024-03-08T13:35:14.338977+0000 mon.smithi119 (mon.0) 502 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7585591] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:17:50 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585591/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585591/ "2024-03-08T13:27:27.835803+0000 mon.smithi149 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585594] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:35:12 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585594/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585594/ "2024-03-08T13:20:00.000230+0000 mon.a (mon.0) 1628 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log [7585595] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:48:09 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585595/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585595/ "2024-03-08T13:30:01.662076+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7585598] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:20:21 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585598/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585598/ "2024-03-08T12:45:31.321971+0000 mon.smithi001 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585599] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:18:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585599/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585599/ "2024-03-08T12:35:28.901769+0000 mon.smithi040 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585601] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:56 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585601/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585601/ "2024-03-08T12:39:59.679889+0000 mon.smithi163 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585603] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:56:59 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585603/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585603/ "2024-03-08T14:10:00.000150+0000 mon.a (mon.0) 1543 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7585604] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:45:12 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585604/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585604/ "2024-03-08T12:42:47.503239+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7585605] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:48 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585605/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585605/ "2024-03-08T13:36:12.307928+0000 mon.smithi117 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585607] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585607/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c5fb851ba884709881db9aae59d1a65 Failed to reconnect to smithi158 [7585608] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:17:54 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585608/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585608/ "2024-03-08T13:37:12.240549+0000 mon.smithi071 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585609] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:38:09 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585609/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585609/ "2024-03-08T12:50:00.000225+0000 mon.a (mon.0) 1161 : cluster 3 [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm" in cluster log [7585611] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:15:22 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585611/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585611/ "2024-03-08T13:34:06.870001+0000 mon.smithi072 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585612] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:19:31 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585612/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585612/ "2024-03-08T12:41:32.243973+0000 mon.smithi164 (mon.0) 250 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585613] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:49:24 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585613/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585613/ "2024-03-08T12:42:24.706412+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7585615] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:19:33 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585615/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585615/ "2024-03-08T13:31:37.299743+0000 mon.smithi003 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585616] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:14:40 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585616/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585616/ "2024-03-08T13:35:42.133523+0000 mon.a (mon.0) 249 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7585617] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:39:58 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585617/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585617/ "2024-03-08T12:43:34.468524+0000 mon.a (mon.0) 678 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7585620] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:18:33 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585620/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585620/ "2024-03-08T13:39:44.067658+0000 mon.smithi067 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585621] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:55:03 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585621/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585621/ "1709905940.5208843 mon.smithi082 (mon.0) 714 : cluster [WRN] Replacing daemon mds.cephfs.smithi082.peyhzf as rank 0 with standby daemon mds.cephfs.smithi082.zktfqy" in cluster log [7585622] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:16:11 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585622/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585622/ "2024-03-08T13:31:01.796785+0000 mon.smithi086 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585624] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:19:25 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585624/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585624/ "2024-03-08T12:38:01.382541+0000 mon.smithi032 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585625] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:50:55 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585625/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585625/ "2024-03-08T13:00:00.000187+0000 mon.a (mon.0) 2724 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log [7585626] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:13:27 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585626/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585626/ "2024-03-08T14:00:00.003643+0000 mon.a (mon.0) 1969 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled" in cluster log [7585628] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:14:52 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585628/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585628/ "2024-03-08T13:31:25.897086+0000 mon.smithi060 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585631] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:16:46 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585631/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585631/ "2024-03-08T13:37:30.372911+0000 mon.smithi012 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585633] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 01:15:18 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585633/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585633/ "2024-03-08T14:10:00.000177+0000 mon.a (mon.0) 963 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7585634] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:21:21 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585634/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585634/ "2024-03-08T12:42:23.402290+0000 mon.smithi154 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585635] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:26 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585635/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585635/ "2024-03-08T12:40:10.104912+0000 mon.smithi055 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585637] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:46 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585637/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585637/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af44bc6b9c2a48fd938446d3ff523321 Command failed (workunit test cephadm/test_repos.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27f99b14d082e0d773119df169ad98479243bcba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7585638] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:58:44 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585638/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585638/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8dc3723f8c54decb0199c504ea76bb2 Command failed (workunit test suites/fsstress.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27f99b14d082e0d773119df169ad98479243bcba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7585640] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:15:50 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585640/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585640/ "2024-03-08T13:30:52.750004+0000 mon.smithi046 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585643] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:52:16 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585643/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585643/ "2024-03-08T13:25:24.203432+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7585645] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:18:44 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585645/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585645/ "2024-03-08T13:30:39.663972+0000 mon.smithi088 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585647] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:57:10 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585647/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585647/ "2024-03-08T14:10:00.000158+0000 mon.a (mon.0) 1050 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7585648] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:18:03 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585648/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585648/ "2024-03-08T14:00:00.000196+0000 mon.a (mon.0) 1869 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering" in cluster log [7585649] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:21:32 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585649/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585649/ "2024-03-08T12:40:33.830332+0000 mon.smithi107 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585652] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:16:26 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585652/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585652/ "2024-03-08T13:18:13.996049+0000 mon.smithi057 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585653] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:20 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585653/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585653/ "2024-03-08T12:41:32.136143+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7585655] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:55:32 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585655/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585655/ "1709907842.0050786 mon.smithi135 (mon.0) 718 : cluster [WRN] Replacing daemon mds.cephfs.smithi135.pfyhhg as rank 0 with standby daemon mds.cephfs.smithi144.ccbdwu" in cluster log [7585657] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:48:09 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585657/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585657/ "2024-03-08T13:40:00.000148+0000 mon.a (mon.0) 1877 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7585660] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:23:35 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585660/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585660/ "2024-03-08T13:24:14.716271+0000 mon.smithi001 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585661] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:16:45 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585661/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585661/ "2024-03-08T14:20:44.639820+0000 mon.smithi026 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585664] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:14:36 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585664/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585664/ "2024-03-08T14:18:29.241190+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7585665] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:37:29 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585665/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585665/ "2024-03-08T13:21:15.473997+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7585667] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:53:41 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585667/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585667/ "2024-03-08T15:00:00.000108+0000 mon.a (mon.0) 1801 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7585668] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:18:02 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585668/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585668/ "2024-03-08T15:01:18.426589+0000 mon.a (mon.0) 173 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7585669] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:17:17 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585669/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585669/ "2024-03-08T14:34:10.566749+0000 mon.smithi189 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585671] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:00:57 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585671/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e00c30c65b34e72a57012de1b86c07d Command failed (workunit test suites/fsstress.sh) on smithi184 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27f99b14d082e0d773119df169ad98479243bcba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7585672] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:19:35 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585672/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585672/ "2024-03-08T13:40:04.123199+0000 mon.smithi032 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585673] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:47:05 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585673/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585673/ "2024-03-08T15:09:07.913869+0000 mon.a (mon.0) 1299 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7585674] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:22 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585674/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585674/ "2024-03-08T14:10:00.000294+0000 mon.a (mon.0) 1030 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7585675] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:26:34 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585675/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585675/ "2024-03-08T14:04:04.771142+0000 mon.smithi077 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585676] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:14:24 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585676/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585676/ "2024-03-08T14:36:34.285064+0000 mon.smithi169 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585677] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 01:17:12 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585677/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585677/ "2024-03-08T15:20:00.000150+0000 mon.a (mon.0) 913 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7585679] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:20:43 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585679/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585679/ "2024-03-08T14:52:19.631389+0000 mon.smithi132 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7585682] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:22:08 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585682/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585682/ "2024-03-08T14:11:31.707899+0000 mon.smithi001 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log Dead ================================================================= [7585651] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585651/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585651/ hit max job timeout [7585656] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585656/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585656/ hit max job timeout [7585663] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585663/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585663/ hit max job timeout Pass ================================================================= [7585584] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:57:40 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585584/ [7585585] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} time: 00:42:06 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585585/ [7585588] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:44:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585588/ [7585589] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} time: 00:28:42 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585589/ [7585590] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:49:06 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585590/ [7585592] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:14:53 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585592/ [7585593] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:26 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585593/ [7585596] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:37:21 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585596/ [7585597] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:50:00 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585597/ [7585600] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:42:29 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585600/ [7585602] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:58 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585602/ [7585606] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli} time: 00:39:46 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585606/ [7585610] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:45:32 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585610/ [7585614] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:36:06 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585614/ [7585618] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:49:08 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585618/ [7585619] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:40:51 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585619/ [7585623] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585623/ [7585627] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:12:18 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585627/ [7585629] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:35:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585629/ [7585630] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:32:17 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585630/ [7585632] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:37:54 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585632/ [7585636] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:47:18 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585636/ [7585639] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} time: 00:28:51 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585639/ [7585641] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} time: 00:15:43 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585641/ [7585642] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:14:43 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585642/ [7585644] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:35:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585644/ [7585646] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:17:45 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585646/ [7585650] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:40:14 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585650/ [7585654] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} time: 00:22:40 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585654/ [7585658] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:33:06 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585658/ [7585659] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:34:29 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585659/ [7585662] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:45:35 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585662/ [7585666] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:25:40 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585666/ [7585670] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:18 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585670/ [7585678] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:39:25 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585678/ [7585680] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:38:05 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585680/ [7585681] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:21:39 info: https://pulpito.ceph.com/gabrioux-2024-03-08_12:13:48-orch:cephadm-wip-guits-testing-6-2024-03-08-0932-squid-distro-default-smithi/7585681/