2024-04-15T20:10:03.023 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T20:10:03.023 INFO:teuthology.results:subset: '25/64' 2024-04-15T20:10:03.023 INFO:teuthology.results:seed: '8042' 2024-04-16T02:19:27.638 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-16T02:19:27.663 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-16T14:20:27.258 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-04-16T14:20:41.265 INFO:scrape:Found 188 jobs 2024-04-16T14:20:59.223 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657333/teuthology.log 2024-04-16T14:21:01.395 INFO:scrape:Found 9 distinct failure reasons 2024-04-16T14:21:01.396 INFO:scrape:Failure: "2024-04-16T01:59:55.494561+0000 mon.smithi151 (mon.0) 860 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log 2024-04-16T14:21:01.396 INFO:scrape:50 jobs 2024-04-16T14:21:01.396 INFO:scrape:suites intersection: [] 2024-04-16T14:21:01.396 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/jaeger', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '3-final}', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/yes', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/nfs/{cluster/{1-node}', 'orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04', 'orch/cephadm/osds/{0-distro/centos_9.stream', 'orch/cephadm/osds/{0-distro/centos_9.stream_runc', 'orch/cephadm/osds/{0-distro/ubuntu_22.04', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke/{0-distro/centos_9.stream', 'orch/cephadm/smoke/{0-distro/ubuntu_22.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch/cephadm/with-work/{0-distro/ubuntu_22.04', 'orch/cephadm/workunits/{0-distro/centos_9.stream', 'orch/cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'roles', 'start', 'start}', 'supported-random-distros$/{centos_latest}', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'task/test_ca_signed_key}', 'task/test_cephadm_timeout}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/{0-from/quincy', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}'] 2024-04-16T14:21:01.397 INFO:scrape: 2024-04-16T14:21:01.397 INFO:scrape:Failure: Command failed on smithi144 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 2024-04-16T14:21:01.397 INFO:scrape:5 jobs: ['7657431', '7657294', '7657248', '7657385', '7657340'] 2024-04-16T14:21:01.397 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_22.04'] 2024-04-16T14:21:01.397 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_22.04', 'rook/1.7.2}', 'rook/master}'] 2024-04-16T14:21:01.397 INFO:scrape: 2024-04-16T14:21:01.397 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-16T14:21:01.397 INFO:scrape:2 jobs: ['7657372', '7657400'] 2024-04-16T14:21:01.398 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-16T14:21:01.398 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-16T14:21:01.398 INFO:scrape: 2024-04-16T14:21:01.398 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T14:21:01.398 INFO:scrape:6 jobs: ['7657351', '7657384', '7657276', '7657306', '7657321', '7657416'] 2024-04-16T14:21:01.398 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}'] 2024-04-16T14:21:01.398 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-16T14:21:01.398 INFO:scrape: 2024-04-16T14:21:01.399 INFO:scrape:Dead: b'2024-04-16T14:19:15.342 DEBUG:teuthology.exit:Finished running handlers' 2024-04-16T14:21:01.399 INFO:scrape:3 jobs: ['7657422', '7657367', '7657312'] 2024-04-16T14:21:01.399 INFO:scrape:suites intersection: [] 2024-04-16T14:21:01.399 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-repo_digest/repo_digest', '2-services/nfs', '3-final}', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity}', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'start}'] 2024-04-16T14:21:01.399 INFO:scrape: 2024-04-16T14:21:01.399 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-04-16T14:21:01.399 INFO:scrape:1 jobs: ['7657303'] 2024-04-16T14:21:01.399 INFO:scrape:suites: ['agent/on', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-04-16T14:21:01.399 INFO:scrape: 2024-04-16T14:21:01.399 INFO:scrape:Failure: Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b97cffaa-fb6a-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' 2024-04-16T14:21:01.399 INFO:scrape:2 jobs: ['7657317', '7657336'] 2024-04-16T14:21:01.400 INFO:scrape:suites intersection: [] 2024-04-16T14:21:01.400 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-repo_digest/repo_digest', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon_election/connectivity}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-16T14:21:01.400 INFO:scrape: 2024-04-16T14:21:01.400 INFO:scrape:Failure: Command failed on smithi167 with status 5: 'sudo systemctl stop ceph-e5ec3232-fb6e-11ee-bc8f-c7b262605968@mon.a' 2024-04-16T14:21:01.400 INFO:scrape:1 jobs: ['7657364'] 2024-04-16T14:21:01.400 INFO:scrape:suites: ['agent/off', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-04-16T14:21:01.400 INFO:scrape: 2024-04-16T14:21:01.400 INFO:scrape:Dead: None 2024-04-16T14:21:01.400 INFO:scrape:1 jobs: ['7657333'] 2024-04-16T14:21:01.400 INFO:scrape: 2024-04-16T14:21:01.407 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/ failed: 67 dead: 3 running: 1 waiting: 0 queued: 0 passed: 117 Fail ================================================================= [7657247] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:24:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657247/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657247/ "2024-04-15T20:41:34.335855+0000 mon.a (mon.0) 439 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657248] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:08:38 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657248/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4717365c403d49f28c1326103e1d97f4 Command failed on smithi083 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7657250] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:17:08 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657250/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657250/ "2024-04-15T20:38:11.915144+0000 mon.smithi062 (mon.0) 824 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657259] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:48:53 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657259/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657259/ "2024-04-15T20:43:25.637243+0000 mon.a (mon.0) 324 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7657263] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:26:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657263/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657263/ "2024-04-15T20:29:24.220858+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7657269] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:28:13 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657269/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657269/ "2024-04-15T20:41:24.553766+0000 mon.a (mon.0) 400 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657274] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:23:49 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657274/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657274/ "2024-04-15T20:45:55.303837+0000 mon.smithi038 (mon.0) 832 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657276] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:12 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657276/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657276/ reached maximum tries (51) after waiting for 300 seconds [7657277] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:16:30 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657277/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657277/ "2024-04-15T20:40:39.519928+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7657281] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:24:07 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657281/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657281/ "2024-04-15T20:41:58.543496+0000 mon.a (mon.0) 257 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657286] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:24:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657286/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657286/ "2024-04-15T20:43:40.228634+0000 mon.a (mon.0) 449 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657288] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:17:32 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657288/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657288/ "2024-04-15T20:37:25.710137+0000 mon.a (mon.0) 357 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657291] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657291/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657291/ "2024-04-15T21:00:00.000108+0000 mon.smithi047 (mon.0) 775 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded" in cluster log [7657292] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:28:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657292/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657292/ "2024-04-15T20:48:27.069016+0000 mon.a (mon.0) 531 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log [7657294] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:08:51 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657294/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657294/ sentry: https://sentry.ceph.com/organizations/ceph/?query=114be81ee8164f6aa20ba7da95ec4c0f Command failed on smithi112 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7657296] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:31:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657296/ "2024-04-15T20:49:02.284881+0000 mon.smithi132 (mon.0) 869 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657297] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:51:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657297/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657297/ "2024-04-15T20:55:51.501729+0000 mon.a (mon.0) 1067 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657299] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:18:51 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657299/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657299/ "2024-04-15T20:38:50.441417+0000 mon.smithi131 (mon.0) 1026 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657300] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:24:23 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657300/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657300/ "2024-04-15T20:46:35.383663+0000 mon.smithi107 (mon.0) 868 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657303] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:10:25 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657303/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9911c6b3f49042688d522d1b57df413c Command failed (workunit test cephadm/test_repos.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7657304] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:25:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657304/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657304/ "2024-04-15T20:43:30.090655+0000 mon.smithi106 (mon.0) 756 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657306] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657306/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657306/ reached maximum tries (51) after waiting for 300 seconds [7657315] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:20:57 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657315/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657315/ "2024-04-15T21:09:54.856923+0000 mon.smithi112 (mon.0) 802 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657317] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:44:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657317/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e5bea7b56f1458eae18cab524e519cb Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b97cffaa- fb6a-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7657319] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:15:50 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657319/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657319/ "2024-04-15T21:06:29.685210+0000 mon.a (mon.0) 471 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657321] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657321/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657321/ reached maximum tries (51) after waiting for 300 seconds [7657325] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:26:35 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657325/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657325/ "2024-04-15T21:10:12.518207+0000 mon.a (mon.0) 385 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657334] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:15:12 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657334/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657334/ "2024-04-15T21:16:12.071713+0000 mon.smithi049 (mon.0) 819 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657335] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:19:14 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657335/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657335/ "2024-04-15T21:12:21.480021+0000 mon.a (mon.0) 383 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657336] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:51:28 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657336/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c8d0f37d5a0422daa93c807014f188a Command failed on smithi066 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9bcc0d0a-fb6c-11ee-bc8f-c7b262605968 -e sha1=87dbfb632e974cb2c28d17423071d2d833b53b29 -- bash -c 'ceph orch ps'" [7657337] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657337/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657337/ "2024-04-15T21:21:47.075752+0000 mon.a (mon.0) 215 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7657340] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:07:17 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657340/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4faf1ce5472a4f72b4c33f097bfdfd8d Command failed on smithi092 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7657342] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:39:36 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657342/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657342/ "2024-04-15T21:21:06.997315+0000 mon.a (mon.0) 458 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657346] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657346/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657346/ "2024-04-15T21:17:58.071544+0000 mon.a (mon.0) 653 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657348] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:22:11 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657348/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657348/ "2024-04-15T21:27:22.343967+0000 mon.smithi082 (mon.0) 972 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657349] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:44:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657349/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657349/ "2024-04-15T21:33:54.965653+0000 mon.a (mon.0) 1077 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657350] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:22:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657350/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657350/ "2024-04-15T21:22:26.389324+0000 mon.a (mon.0) 437 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657351] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657351/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657351/ reached maximum tries (51) after waiting for 300 seconds [7657353] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:35:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657353/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657353/ "2024-04-15T21:24:20.004196+0000 mon.a (mon.0) 612 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657354] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:27 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657354/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657354/ "2024-04-15T21:23:45.667486+0000 mon.a (mon.0) 540 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log [7657357] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:24:48 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657357/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657357/ "2024-04-15T21:21:09.282802+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7657359] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:48 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657359/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657359/ "2024-04-15T21:26:57.680656+0000 mon.smithi087 (mon.0) 847 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657364] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:12 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657364/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6c8192157e8490998726ff690f5d8bb Command failed on smithi167 with status 5: 'sudo systemctl stop ceph-e5ec3232-fb6e-11ee-bc8f-c7b262605968@mon.a' [7657365] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:21:20 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657365/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657365/ "2024-04-16T01:59:55.494561+0000 mon.smithi151 (mon.0) 860 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657369] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:47:53 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657369/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657369/ "2024-04-16T02:07:20.182464+0000 mon.a (mon.0) 998 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657372] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:38:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657372/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79dc9ad203d94c77a69af4334d7b862e Command failed (workunit test suites/fsstress.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7657379] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657379/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657379/ "2024-04-16T02:05:38.946197+0000 mon.smithi042 (mon.0) 830 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657381] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:12:36 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657381/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657381/ "2024-04-16T02:06:44.675065+0000 mon.a (mon.0) 451 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657383] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:19:33 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657383/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657383/ "2024-04-16T02:10:59.080908+0000 mon.a (mon.0) 342 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657384] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:07:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657384/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657384/ reached maximum tries (51) after waiting for 300 seconds [7657385] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/1.7.2} ----------------------------------------------------------------- time: 00:07:32 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657385/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=137b08a589764babbd26e48cb4521f32 Command failed on smithi055 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7657393] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:15:24 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657393/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657393/ "2024-04-16T02:16:55.868369+0000 mon.a (mon.0) 773 : cluster [WRN] Health check failed: 1 stray host(s) with 1 daemon(s) not managed by cephadm (CEPHADM_STRAY_HOST)" in cluster log [7657394] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:25:26 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657394/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657394/ "2024-04-16T02:15:15.873713+0000 mon.a (mon.0) 456 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657398] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:22:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657398/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657398/ "2024-04-16T02:23:50.936223+0000 mon.smithi135 (mon.0) 789 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657399] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657399/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657399/ "2024-04-16T02:15:43.324131+0000 mon.a (mon.0) 183 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657400] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:57 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657400/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59ae7626543b4b169aae27794c65b8b3 Command failed (workunit test suites/fsstress.sh) on smithi079 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7db60e5ba4c4d053b594b7ec58703d2833c55b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7657401] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:13:56 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657401/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657401/ "2024-04-16T02:20:23.338393+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7657406] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:43 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657406/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657406/ "2024-04-16T02:20:01.945835+0000 mon.a (mon.0) 452 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657412] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:21:17 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657412/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657412/ "2024-04-16T02:34:07.724855+0000 mon.smithi139 (mon.0) 941 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657415] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657415/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657415/ "2024-04-16T02:27:59.918863+0000 mon.a (mon.0) 293 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7657416] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:07:17 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657416/ reached maximum tries (51) after waiting for 300 seconds [7657417] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657417/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657417/ "2024-04-16T02:29:23.831753+0000 mon.a (mon.0) 533 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log [7657420] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:44 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657420/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657420/ "2024-04-16T02:32:33.910942+0000 mon.smithi087 (mon.0) 852 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657425] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:14:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657425/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657425/ "2024-04-16T02:35:37.095599+0000 mon.smithi076 (mon.0) 842 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657429] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:50 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657429/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657429/ "2024-04-16T02:41:11.556311+0000 mon.smithi059 (mon.0) 790 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7657430] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657430/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657430/ "2024-04-16T02:38:13.556335+0000 mon.a (mon.0) 341 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7657431] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/master} ----------------------------------------------------------------- time: 00:07:19 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657431/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657431/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba8ea358d27e4d70835a53381011238a Command failed on smithi144 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" Dead ================================================================= [7657312] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657312/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657312/ hit max job timeout [7657333] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657333/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657333/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7657367] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657367/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657367/ hit max job timeout Running ================================================================= [7657422] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657422/ Pass ================================================================= [7657244] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} time: 00:44:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657244/ [7657245] orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} time: 00:28:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657245/ [7657246] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:24:48 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657246/ [7657249] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:14:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657249/ [7657251] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:18:29 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657251/ [7657252] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:27:14 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657252/ [7657253] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:18:25 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657253/ [7657254] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:52:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657254/ [7657255] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:29:49 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657255/ [7657256] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:24:55 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657256/ [7657257] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657257/ [7657258] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:35:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657258/ [7657260] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption} time: 00:18:36 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657260/ [7657261] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} time: 00:13:11 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657261/ [7657262] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw} time: 00:13:29 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657262/ [7657264] orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} time: 00:20:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657264/ [7657265] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:24:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657265/ [7657266] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} time: 00:11:57 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657266/ [7657267] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:21:36 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657267/ [7657268] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:39:37 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657268/ [7657270] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:25:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657270/ [7657271] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:30:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657271/ [7657272] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:22:29 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657272/ [7657273] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite} time: 00:25:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657273/ [7657275] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:40:08 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657275/ [7657278] orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} time: 00:20:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657278/ [7657279] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:18:09 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657279/ [7657280] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:31:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657280/ [7657282] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:23:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657282/ [7657283] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:22:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657283/ [7657284] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:21:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657284/ [7657285] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:22:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657285/ [7657287] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:39:18 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657287/ [7657289] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657289/ [7657290] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:55:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657290/ [7657293] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:21:48 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657293/ [7657295] orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} time: 00:26:37 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657295/ [7657298] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_cephadm} time: 00:22:30 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657298/ [7657301] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657301/ [7657302] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:21:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657302/ [7657305] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:37:40 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657305/ [7657307] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli_mon} time: 00:23:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657307/ [7657308] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} time: 00:14:07 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657308/ [7657309] orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} time: 00:11:09 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657309/ [7657310] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657310/ [7657311] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} time: 00:19:29 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657311/ [7657313] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:14:41 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657313/ [7657314] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657314/ [7657316] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:23:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657316/ [7657318] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:25:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657318/ [7657320] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657320/ [7657322] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_adoption} time: 00:10:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657322/ [7657323] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw} time: 00:19:33 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657323/ [7657324] orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} time: 00:13:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657324/ [7657326] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657326/ [7657327] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:05:32 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657327/ [7657328] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:20:56 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657328/ [7657329] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657329/ [7657330] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:15:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657330/ [7657331] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:12:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657331/ [7657332] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:23:42 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657332/ [7657338] orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} time: 00:18:27 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657338/ [7657339] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:20:07 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657339/ [7657341] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:46:52 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657341/ [7657343] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:15:32 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657343/ [7657344] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:13:35 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657344/ [7657345] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657345/ [7657347] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:15:37 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657347/ [7657352] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:35:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657352/ [7657355] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} time: 00:14:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657355/ [7657356] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:19:42 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657356/ [7657358] orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} time: 00:12:44 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657358/ [7657360] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} time: 00:11:29 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657360/ [7657361] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:11:24 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657361/ [7657362] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:38:36 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657362/ [7657363] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657363/ [7657366] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:13 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657366/ [7657368] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_cephadm} time: 00:16:44 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657368/ [7657370] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:55 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657370/ [7657371] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:22:41 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657371/ [7657373] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli_mon} time: 00:24:56 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657373/ [7657374] orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} time: 00:11:18 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657374/ [7657375] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:49 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657375/ [7657376] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:35:44 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657376/ [7657377] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:20:05 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657377/ [7657378] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_cephadm_repos} time: 00:06:30 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657378/ [7657380] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:59 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657380/ [7657382] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:16:25 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657382/ [7657386] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_adoption} time: 00:12:28 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657386/ [7657387] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} time: 00:14:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657387/ [7657388] orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} time: 00:19:17 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657388/ [7657389] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:43 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657389/ [7657390] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:25:02 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657390/ [7657391] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:13:38 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657391/ [7657392] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:52:54 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657392/ [7657395] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:14:50 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657395/ [7657396] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:57:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657396/ [7657397] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:19:05 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657397/ [7657402] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} time: 00:11:45 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657402/ [7657403] orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} time: 00:11:12 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657403/ [7657404] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657404/ [7657405] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} time: 00:12:50 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657405/ [7657407] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:15:33 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657407/ [7657408] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657408/ [7657409] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:44:15 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657409/ [7657410] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:35:44 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657410/ [7657411] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_monitoring_stack_basic} time: 00:25:11 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657411/ [7657413] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:15:16 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657413/ [7657414] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:40 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657414/ [7657418] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657418/ [7657419] orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} time: 00:12:57 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657419/ [7657421] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:24:18 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657421/ [7657423] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:53 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657423/ [7657424] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:37:40 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657424/ [7657426] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:11:04 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657426/ [7657427] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:32:48 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657427/ [7657428] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:11 info: https://pulpito.ceph.com/teuthology-2024-04-15_20:08:15-orch-main-distro-default-smithi/7657428/