2024-02-05T13:10:44.737 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-05T13:10:44.738 INFO:teuthology.results:subset: '1/8' 2024-02-05T13:10:44.738 INFO:teuthology.results:seed: '8480' 2024-02-05T15:56:28.399 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-05T15:56:28.422 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-06T03:50:27.131 INFO:teuthology.results:Tests finished! gathering results... 2024-02-06T03:50:34.925 INFO:scrape:Found 99 jobs 2024-02-06T03:50:57.939 INFO:scrape:Found 8 distinct failure reasons 2024-02-06T03:50:57.939 INFO:scrape:Failure: "2024-02-05T16:10:00.068708+0000 mon.a (mon.0) 664 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log 2024-02-06T03:50:57.939 INFO:scrape:63 jobs 2024-02-06T03:50:57.940 INFO:scrape:suites intersection: [] 2024-02-06T03:50:57.940 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_set_mon_crush_locations}', 'tasks/rados_api_tests}', 'tasks/rados_python}'] 2024-02-06T03:50:57.940 INFO:scrape: 2024-02-06T03:50:57.940 INFO:scrape:Dead: b'2024-02-06T03:47:16.903 DEBUG:teuthology.exit:Finished running handlers' 2024-02-06T03:50:57.940 INFO:scrape:1 jobs: ['7546887'] 2024-02-06T03:50:57.941 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-06T03:50:57.941 INFO:scrape: 2024-02-06T03:50:57.941 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-06T03:50:57.941 INFO:scrape:2 jobs: ['7546854', '7546823'] 2024-02-06T03:50:57.941 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-06T03:50:57.941 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-06T03:50:57.941 INFO:scrape: 2024-02-06T03:50:57.941 INFO:scrape:Failure: "1707143447.0918276 mon.smithi084 (mon.0) 709 : cluster [WRN] Replacing daemon mds.cephfs.smithi084.tsafew as rank 0 with standby daemon mds.cephfs.smithi084.mlpbir" in cluster log 2024-02-06T03:50:57.941 INFO:scrape:3 jobs: ['7546837', '7546802', '7546871'] 2024-02-06T03:50:57.942 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-06T03:50:57.942 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-06T03:50:57.942 INFO:scrape: 2024-02-06T03:50:57.942 INFO:scrape:Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2024-02-06T03:50:57.942 INFO:scrape:1 jobs: ['7546886'] 2024-02-06T03:50:57.942 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_orch_cli}'] 2024-02-06T03:50:57.942 INFO:scrape: 2024-02-06T03:50:57.942 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6126c574839efbfa410f337a8812351fb1f36293 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-02-06T03:50:57.942 INFO:scrape:1 jobs: ['7546853'] 2024-02-06T03:50:57.942 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-02-06T03:50:57.942 INFO:scrape: 2024-02-06T03:50:57.943 INFO:scrape:Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6486 (apt-get)'] 2024-02-06T03:50:57.943 INFO:scrape:1 jobs: ['7546846'] 2024-02-06T03:50:57.943 INFO:scrape:suites: ['fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'start', 'tasks/rados_python}'] 2024-02-06T03:50:57.943 INFO:scrape: 2024-02-06T03:50:57.943 INFO:scrape:Failure: "2024-02-05T13:31:37.910814+0000 mon.smithi084 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-02-06T03:50:57.943 INFO:scrape:2 jobs: ['7546803', '7546811'] 2024-02-06T03:50:57.943 INFO:scrape:suites intersection: [] 2024-02-06T03:50:57.943 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '3-upgrade-with-workload', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'mon_election/classic}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream'] 2024-02-06T03:50:57.943 INFO:scrape: 2024-02-06T03:50:57.947 INFO:teuthology.results:Sending results to gabrioux@ibm.com: Test Run: gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/ failed: 72 dead: 2 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [7546800] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:03 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546800/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546800/ "2024-02-05T13:40:00.000258+0000 mon.a (mon.0) 1248 : cluster 3 [WRN] OSDMAP_FLAGS: noup flag(s) set" in cluster log [7546802] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:50 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546802/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546802/ "1707140122.5807512 mon.smithi016 (mon.0) 684 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7546803] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:00 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546803/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546803/ "2024-02-05T13:31:37.910814+0000 mon.smithi084 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7546807] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:28 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546807/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546807/ "2024-02-05T13:34:07.167335+0000 mon.smithi079 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546809] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:19 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546809/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546809/ "2024-02-05T13:39:07.564729+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546810] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:51:30 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546810/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546810/ "2024-02-05T14:00:00.000138+0000 mon.a (mon.0) 1667 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7546811] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:48:09 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546811/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546811/ "2024-02-05T13:42:41.436414+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7546814] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:16:07 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546814/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546814/ "2024-02-05T13:41:46.905468+0000 mon.smithi106 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546815] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:15 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546815/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546815/ "2024-02-05T13:48:49.509832+0000 mon.smithi026 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546816] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:18:28 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546816/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546816/ "2024-02-05T13:49:23.248236+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546817] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:56 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546817/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546817/ "2024-02-05T13:57:05.420746+0000 mon.smithi123 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546818] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:25:00 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546818/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546818/ "2024-02-05T13:56:46.982228+0000 mon.a (mon.0) 234 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7546819] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:29:41 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546819/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546819/ "2024-02-05T14:10:00.000145+0000 mon.a (mon.0) 1657 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7546820] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:00:59 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546820/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546820/ "2024-02-05T14:20:00.000235+0000 mon.a (mon.0) 2507 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log [7546821] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:24 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546821/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546821/ "2024-02-05T13:56:21.461914+0000 mon.smithi094 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546823] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:01 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546823/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fa064a8f4b34b208347f8abd5c3281c Command failed (workunit test suites/fsstress.sh) on smithi157 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6126c574839efbfa410f337a8812351fb1f36293 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7546824] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:13:49 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546824/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546824/ "2024-02-05T14:02:18.264515+0000 mon.smithi007 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546825] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:38:08 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546825/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546825/ "2024-02-05T14:17:17.760971+0000 mon.a (mon.0) 379 : cluster 3 [WRN] MON_DOWN: 1/4 mons down, quorum a,e,c" in cluster log [7546826] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:12 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546826/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546826/ "2024-02-05T14:20:00.000139+0000 mon.a (mon.0) 1178 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log [7546827] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546827/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546827/ "2024-02-05T14:11:18.089296+0000 mon.smithi079 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546828] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:19:48 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546828/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546828/ "2024-02-05T14:13:00.796386+0000 mon.smithi133 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546829] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:48:04 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546829/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546829/ "2024-02-05T14:14:01.551595+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7546831] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:18:26 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546831/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546831/ "2024-02-05T14:18:49.607723+0000 mon.smithi029 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546832] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:12:08 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546832/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546832/ "2024-02-05T14:24:41.496139+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546833] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:40:19 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546833/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546833/ "2024-02-05T14:27:11.338851+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7546836] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:15:30 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546836/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546836/ "2024-02-05T14:24:31.629562+0000 mon.smithi123 (mon.0) 266 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546837] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:15 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546837/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546837/ "1707143447.0918276 mon.smithi084 (mon.0) 709 : cluster [WRN] Replacing daemon mds.cephfs.smithi084.tsafew as rank 0 with standby daemon mds.cephfs.smithi084.mlpbir" in cluster log [7546838] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:24 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546838/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546838/ "2024-02-05T14:25:23.996331+0000 mon.smithi022 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546839] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546839/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546839/ "2024-02-05T14:30:00.148763+0000 mon.a (mon.0) 800 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7546840] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:18:59 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546840/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546840/ "2024-02-05T14:29:06.715299+0000 mon.smithi080 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546841] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:03:49 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546841/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546841/ "2024-02-05T14:40:27.117579+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7546842] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:51:11 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546842/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546842/ "2024-02-05T14:50:00.000386+0000 mon.a (mon.0) 1618 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled" in cluster log [7546844] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546844/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546844/ "2024-02-05T14:40:55.271676+0000 mon.smithi133 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546847] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:14:22 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546847/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546847/ "2024-02-05T14:49:47.623087+0000 mon.smithi042 (mon.0) 266 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546849] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:57:27 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546849/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546849/ "2024-02-05T15:10:00.000318+0000 mon.a (mon.0) 1099 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7546850] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:22:06 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546850/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546850/ "2024-02-05T15:00:03.612170+0000 mon.smithi120 (mon.0) 257 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546851] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:57 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546851/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546851/ "2024-02-05T14:59:30.007951+0000 mon.smithi094 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546853] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:43 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546853/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca9c49da38ea4a5ea48e3c84978e7450 Command failed (workunit test cephadm/test_repos.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6126c574839efbfa410f337a8812351fb1f36293 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7546854] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:01 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546854/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a2b944a82de435c9adbd11fd6c15f92 Command failed (workunit test suites/fsstress.sh) on smithi119 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6126c574839efbfa410f337a8812351fb1f36293 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7546856] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:14:26 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546856/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546856/ "2024-02-05T14:57:33.713788+0000 mon.smithi029 (mon.0) 267 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546858] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:36 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546858/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546858/ "2024-02-05T14:57:16.204187+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546859] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:52:47 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546859/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546859/ "2024-02-05T15:01:17.433154+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7546861] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:15:20 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546861/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546861/ "2024-02-05T15:12:05.165471+0000 mon.smithi016 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546863] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:31:11 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546863/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546863/ "2024-02-05T15:20:00.000137+0000 mon.a (mon.0) 917 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7546864] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:51:05 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546864/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546864/ "2024-02-05T15:20:00.084010+0000 mon.a (mon.0) 1406 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7546865] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:21:59 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546865/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546865/ "2024-02-05T15:15:05.363997+0000 mon.smithi084 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546866] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:13:18 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546866/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546866/ "2024-02-05T15:15:22.258822+0000 mon.a (mon.0) 23 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log [7546867] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:20 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546867/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546867/ "2024-02-05T15:11:13.097650+0000 mon.smithi106 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546868] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:14:50 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546868/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546868/ "2024-02-05T15:13:14.274712+0000 mon.smithi113 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546870] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:22:23 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546870/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546870/ "2024-02-05T15:24:40.321466+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546871] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:33 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546871/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546871/ "1707147149.398673 mon.smithi144 (mon.0) 721 : cluster [WRN] Replacing daemon mds.cephfs.smithi144.ghrfxi as rank 0 with standby daemon mds.cephfs.smithi179.riukmo" in cluster log [7546872] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:06 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546872/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546872/ "2024-02-05T15:26:16.471822+0000 mon.smithi043 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546873] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:49:44 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546873/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546873/ "2024-02-05T15:50:00.000305+0000 mon.a (mon.0) 1422 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7546876] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:23:55 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546876/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546876/ "2024-02-05T15:30:43.549320+0000 mon.smithi042 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546877] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:36 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546877/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546877/ "2024-02-05T15:28:25.552718+0000 mon.smithi029 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546878] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:18:05 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546878/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546878/ "2024-02-05T15:37:55.344996+0000 mon.a (mon.0) 579 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7546879] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:15:13 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546879/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546879/ "2024-02-05T15:36:43.810119+0000 mon.smithi070 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546880] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:31 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546880/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546880/ "2024-02-05T15:41:45.954876+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546881] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:37:27 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546881/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546881/ "2024-02-05T15:43:48.306738+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7546883] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:42:54 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546883/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546883/ "2024-02-05T15:50:00.000169+0000 mon.a (mon.0) 1118 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7546884] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:34:55 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546884/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546884/ "2024-02-05T15:50:00.497536+0000 mon.a (mon.0) 1115 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7546885] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:03 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546885/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546885/ "2024-02-05T15:50:54.045520+0000 mon.smithi106 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546886] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:19:01 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546886/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546886/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b3d7423be6f411bb58e680cb0580aa4 Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7546888] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:20:02 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546888/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546888/ "2024-02-05T16:01:39.716945+0000 mon.smithi043 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546889] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:23:26 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546889/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546889/ "2024-02-05T16:01:04.691095+0000 mon.a (mon.0) 272 : cluster 3 [WRN] MON_DOWN: 1/5 mons down, quorum a,e,c,d" in cluster log [7546890] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:23 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546890/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546890/ "2024-02-05T16:10:00.068708+0000 mon.a (mon.0) 664 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7546891] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:33 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546891/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546891/ "2024-02-05T16:03:21.083762+0000 mon.smithi039 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546892] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:14:44 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546892/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546892/ "2024-02-05T15:58:37.880385+0000 mon.smithi029 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546893] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:50:26 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546893/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546893/ "2024-02-05T16:10:00.000107+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7546895] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:17:12 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546895/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546895/ "2024-02-05T16:14:03.837837+0000 mon.smithi022 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7546897] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546897/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546897/ "2024-02-05T16:18:08.152372+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7546898] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:21:48 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546898/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546898/ "2024-02-05T16:16:48.529649+0000 mon.smithi042 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log Dead ================================================================= [7546846] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:06:54 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546846/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8e3370b10734a1d9823fa319b639e04 ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6486 (apt-get)'] [7546887] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546887/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546887/ hit max job timeout Pass ================================================================= [7546801] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} time: 00:15:17 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546801/ [7546804] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:44:28 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546804/ [7546805] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} time: 00:14:00 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546805/ [7546806] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:25:17 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546806/ [7546808] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} time: 00:11:17 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546808/ [7546812] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:03 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546812/ [7546813] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:23:52 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546813/ [7546822] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli} time: 00:12:56 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546822/ [7546830] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:12:47 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546830/ [7546834] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:23:21 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546834/ [7546835] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:21 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546835/ [7546843] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:12:14 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546843/ [7546845] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:11:22 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546845/ [7546848] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:14:13 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546848/ [7546852] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:23:09 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546852/ [7546855] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} time: 00:13:57 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546855/ [7546857] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} time: 00:12:43 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546857/ [7546860] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:13:33 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546860/ [7546862] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:55 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546862/ [7546869] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:32:23 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546869/ [7546874] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:41 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546874/ [7546875] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:23:17 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546875/ [7546882] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:25:06 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546882/ [7546894] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:13:47 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546894/ [7546896] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:23:17 info: https://pulpito.ceph.com/gabrioux-2024-02-05_13:08:40-orch:cephadm-wip-guits-testing-3-2024-02-05-0946-distro-default-smithi/7546896/