2024-04-08T21:13:11.523 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-08T21:13:11.523 INFO:teuthology.results:subset: '4/64' 2024-04-08T21:13:11.523 INFO:teuthology.results:seed: '1383' 2024-04-10T08:31:43.394 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-10T08:31:43.420 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-10T09:17:47.535 INFO:teuthology.results:Tests finished! gathering results... 2024-04-10T09:18:00.784 INFO:scrape:Found 172 jobs 2024-04-10T09:18:04.357 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:04.357 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:04.358 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:04.438 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:04.438 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:04.439 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:07.813 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646341/teuthology.log 2024-04-10T09:18:24.655 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:24.655 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:24.655 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:24.684 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:24.684 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:24.684 WARNING:scrape:Saw end of BT but not start 2024-04-10T09:18:31.638 INFO:scrape:Found 12 distinct failure reasons 2024-04-10T09:18:31.639 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-04-10T09:18:31.639 INFO:scrape:1 jobs: ['7646223'] 2024-04-10T09:18:31.639 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/rados_api_tests', 'fixed-2', 'msgr/async-v1only', 'orch/cephadm/thrash/{0-distro/ubuntu_22.04', 'root}'] 2024-04-10T09:18:31.639 INFO:scrape: 2024-04-10T09:18:31.639 INFO:scrape:Failure: "2024-04-08T23:12:18.586458+0000 mon.a (mon.0) 483 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-04-10T09:18:31.639 INFO:scrape:31 jobs 2024-04-10T09:18:31.640 INFO:scrape:suites intersection: [] 2024-04-10T09:18:31.640 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'conf/{disable-pool-app}', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke/{0-distro/centos_9.stream', 'orch/cephadm/thrash/{0-distro/centos_9.stream', 'orch/cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/ubuntu_22.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch/cephadm/with-work/{0-distro/centos_9.stream', 'orch/cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch/cephadm/with-work/{0-distro/ubuntu_22.04', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'task/test_host_drain}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'workloads/cephadm_iscsi}'] 2024-04-10T09:18:31.640 INFO:scrape: 2024-04-10T09:18:31.640 INFO:scrape:Failure: Command failed on smithi096 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 2024-04-10T09:18:31.640 INFO:scrape:4 jobs: ['7646278', '7646231', '7646189', '7646319'] 2024-04-10T09:18:31.640 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_22.04'] 2024-04-10T09:18:31.640 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_22.04', 'rook/1.7.2}', 'rook/master}'] 2024-04-10T09:18:31.640 INFO:scrape: 2024-04-10T09:18:31.640 INFO:scrape:Dead: b'2024-04-09T10:23:31.770 DEBUG:teuthology.exit:Finished running handlers' 2024-04-10T09:18:31.641 INFO:scrape:2 jobs: ['7646182', '7646190'] 2024-04-10T09:18:31.641 INFO:scrape:suites intersection: [] 2024-04-10T09:18:31.641 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-services/client-keyring', '3-final}', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-10T09:18:31.641 INFO:scrape: 2024-04-10T09:18:31.641 INFO:scrape:Failure: "2024-04-08T23:20:00.000151+0000 mon.a (mon.0) 1193 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull" in cluster log 2024-04-10T09:18:31.641 INFO:scrape:1 jobs: ['7646237'] 2024-04-10T09:18:31.641 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/radosbench', 'fixed-2', 'msgr/async-v2only', 'orch/cephadm/thrash/{0-distro/centos_9.stream', 'root}'] 2024-04-10T09:18:31.641 INFO:scrape: 2024-04-10T09:18:31.641 INFO:scrape:Dead: None 2024-04-10T09:18:31.642 INFO:scrape:1 jobs: ['7646341'] 2024-04-10T09:18:31.642 INFO:scrape: 2024-04-10T09:18:31.642 INFO:scrape:Failure: Command failed on smithi121 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a0fdd25c-f70e-11ee-b64a-cb9ed24678a4 -e sha1=b6f3e469c8285b60ad2826d6f524be2e6014ed05 -- bash -c 'ceph orch ps'" 2024-04-10T09:18:31.642 INFO:scrape:3 jobs: ['7646282', '7646226', '7646302'] 2024-04-10T09:18:31.642 INFO:scrape:suites intersection: [] 2024-04-10T09:18:31.642 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-repo_digest/repo_digest', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-10T09:18:31.642 INFO:scrape: 2024-04-10T09:18:31.642 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-10T09:18:31.642 INFO:scrape:6 jobs: ['7646308', '7646230', '7646261', '7646336', '7646169', '7646204'] 2024-04-10T09:18:31.643 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-04-10T09:18:31.643 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-10T09:18:31.643 INFO:scrape: 2024-04-10T09:18:31.643 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-10T09:18:31.643 INFO:scrape:2 jobs: ['7646216', '7646245'] 2024-04-10T09:18:31.643 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-10T09:18:31.643 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-10T09:18:31.643 INFO:scrape: 2024-04-10T09:18:31.644 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-10T09:18:31.644 INFO:scrape:2 jobs: ['7646213', '7646288'] 2024-04-10T09:18:31.644 INFO:scrape:suites intersection: ['agent/on', 'task/test_cephadm}'] 2024-04-10T09:18:31.644 INFO:scrape:suites union: ['agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-04-10T09:18:31.644 INFO:scrape: 2024-04-10T09:18:31.644 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-04-10T09:18:31.644 INFO:scrape:1 jobs: ['7646294'] 2024-04-10T09:18:31.644 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-04-10T09:18:31.644 INFO:scrape: 2024-04-10T09:18:31.644 INFO:scrape:Failure: Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-3e16b038-f717-11ee-b64a-cb9ed24678a4@mon.a' 2024-04-10T09:18:31.644 INFO:scrape:1 jobs: ['7646342'] 2024-04-10T09:18:31.645 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-04-10T09:18:31.645 INFO:scrape: 2024-04-10T09:18:31.652 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/ failed: 52 dead: 3 running: 0 waiting: 0 queued: 0 passed: 117 Fail ================================================================= [7646169] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:36 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646169/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646169/ reached maximum tries (51) after waiting for 300 seconds [7646170] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646170/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646170/ "2024-04-08T21:39:47.645798+0000 mon.smithi102 (mon.0) 870 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7646175] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:25:38 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646175/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646175/ "2024-04-08T22:35:09.552246+0000 mon.a (mon.0) 208 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646179] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:23 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646179/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646179/ "2024-04-08T22:35:04.073844+0000 mon.a (mon.0) 208 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646180] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:03:22 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646180/ "2024-04-08T22:36:14.134517+0000 mon.a (mon.0) 198 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7646181] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:56:31 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646181/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646181/ "2024-04-08T22:51:07.971223+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7646189] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/host rook/master} ----------------------------------------------------------------- time: 00:07:53 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646189/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4760afc2ec24b449e7ae59a3319e691 Command failed on smithi023 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7646198] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:16:38 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646198/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646198/ "2024-04-08T22:41:44.542237+0000 mon.a (mon.0) 268 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646200] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:32:37 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646200/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646200/ "2024-04-08T22:36:18.030469+0000 mon.a (mon.0) 359 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7646202] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:47:57 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646202/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646202/ "2024-04-08T22:44:05.998219+0000 mon.a (mon.0) 201 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7646204] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:57 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646204/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646204/ reached maximum tries (51) after waiting for 300 seconds [7646207] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:50:40 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646207/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646207/ "2024-04-08T22:43:00.083325+0000 mon.a (mon.0) 198 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7646213] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:16:49 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646213/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3f8d1762e7048b78fd6851ab36d893f Command failed (workunit test cephadm/test_cephadm.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7646215] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:29 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646215/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646215/ "2024-04-08T23:00:00.000126+0000 mon.a (mon.0) 1219 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log [7646216] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:44 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646216/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2c61a1a744a4c32b6f777c856c1e8af Command failed (workunit test suites/fsstress.sh) on smithi120 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7646221] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:05 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646221/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646221/ "2024-04-08T22:59:05.001191+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646223] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:04:48 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646223/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63545c2427db4b28b2645eca5ae2018e Command failed (workunit test rados/test.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7646226] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:50:28 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646226/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67814a907da848d0ad92b7dddaf03f1b Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa01dac4-f5fb-11ee-b649-cb9ed24678a4 -e sha1=b6f3e469c8285b60ad2826d6f524be2e6014ed05 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7646230] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:09:43 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646230/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646230/ reached maximum tries (51) after waiting for 300 seconds [7646231] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:07:35 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646231/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646231/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fb0896279214dd5918edff9089045e8 Command failed on smithi090 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7646237] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 01:01:06 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646237/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646237/ "2024-04-08T23:20:00.000151+0000 mon.a (mon.0) 1193 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull" in cluster log [7646238] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:22:43 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646238/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646238/ "2024-04-08T23:20:00.000211+0000 mon.a (mon.0) 645 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7646241] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:50 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646241/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646241/ "2024-04-08T23:07:21.876894+0000 mon.a (mon.0) 209 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646245] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:32 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646245/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646245/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8f33d0eae96437d9b9acad708aa0da6 Command failed (workunit test suites/fsstress.sh) on smithi171 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7646248] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:28:34 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646248/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646248/ "2024-04-08T23:12:18.586458+0000 mon.a (mon.0) 483 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7646250] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:30:43 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646250/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646250/ "2024-04-08T23:20:00.000125+0000 mon.a (mon.0) 868 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7646251] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:52:23 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646251/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646251/ "2024-04-08T23:30:00.000230+0000 mon.a (mon.0) 2695 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled" in cluster log [7646261] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:07:50 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646261/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646261/ reached maximum tries (51) after waiting for 300 seconds [7646263] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:31 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646263/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646263/ "2024-04-08T23:21:17.507538+0000 mon.smithi033 (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7646268] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:25:18 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646268/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646268/ "2024-04-08T23:25:39.394430+0000 mon.a (mon.0) 210 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646275] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:50:57 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646275/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646275/ "2024-04-08T23:50:00.000164+0000 mon.a (mon.0) 1242 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7646277] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:43 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646277/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646277/ "2024-04-08T23:37:36.746462+0000 mon.a (mon.0) 200 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7646278] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:07:17 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646278/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646278/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac63dd0a78d747ab9f346eb2be914045 Command failed on smithi096 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7646280] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:28:53 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646280/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646280/ "2024-04-10T07:50:38.914922+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7646282] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:14 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646282/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646282/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d7fd7adae3d4dbb8514ddfb1f0f4cbf Command failed on smithi121 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a0fdd25c-f70e-11ee-b64a-cb9ed24678a4 -e sha1=b6f3e469c8285b60ad2826d6f524be2e6014ed05 -- bash -c 'ceph orch ps'" [7646288] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:13:26 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646288/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f24763647f144d29e131893d6af58e3 Command failed (workunit test cephadm/test_cephadm.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7646289] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:37:58 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646289/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646289/ "2024-04-10T08:10:00.000140+0000 mon.a (mon.0) 2265 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7646291] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:12:48 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646291/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646291/ "2024-04-10T07:59:05.560684+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646294] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:09:04 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646294/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646294/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02dfb38b770e42dba42435348bd62b81 Command failed (workunit test cephadm/test_repos.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4870d5a890743a871e99c190e2baea5ccf0df349 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7646301] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:53:25 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646301/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646301/ "2024-04-10T08:20:00.000127+0000 mon.a (mon.0) 1067 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7646302] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:49:37 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646302/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0578d98b259943a2955ea328d2f2065d Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 30c3d2ae-f711-11ee-b64a-cb9ed24678a4 -e sha1=b6f3e469c8285b60ad2826d6f524be2e6014ed05 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7646303] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:51 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646303/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646303/ "2024-04-10T08:10:00.003182+0000 mon.a (mon.0) 834 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7646306] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:51:44 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646306/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646306/ "2024-04-10T00:50:00.161024+0000 mon.a (mon.0) 1254 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7646308] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:56 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646308/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646308/ reached maximum tries (51) after waiting for 300 seconds [7646313] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:51 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646313/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646313/ "2024-04-10T08:14:07.707564+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7646318] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:43:19 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646318/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646318/ "2024-04-10T08:40:00.000145+0000 mon.a (mon.0) 1535 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7646319] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:07:24 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646319/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f801b06634634e7683c501a525378416 Command failed on smithi122 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7646325] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:36 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646325/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646325/ "2024-04-10T08:41:12.728046+0000 mon.a (mon.0) 201 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7646328] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:31:41 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646328/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646328/ "2024-04-10T08:38:59.079227+0000 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7646333] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:37:09 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646333/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646333/ "2024-04-10T09:00:00.000193+0000 mon.a (mon.0) 1660 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7646336] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:44 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646336/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646336/ reached maximum tries (51) after waiting for 300 seconds [7646342] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:12:01 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646342/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=475ea0183b92424c8ddfad85cc2133a8 Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-3e16b038-f717-11ee-b64a-cb9ed24678a4@mon.a' Dead ================================================================= [7646182] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646182/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646182/ hit max job timeout [7646190] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646190/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646190/ hit max job timeout [7646341] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646341/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646341/ Error reimaging machines: Failed to power on smithi174 Pass ================================================================= [7646166] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:18:28 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646166/ [7646167] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_host_drain} time: 00:16:03 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646167/ [7646168] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:22:08 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646168/ [7646171] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:45:00 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646171/ [7646172] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption} time: 00:10:34 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646172/ [7646173] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} time: 00:14:13 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646173/ [7646174] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:31 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646174/ [7646176] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:29 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646176/ [7646177] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:13:19 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646177/ [7646178] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:52 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646178/ [7646183] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:31 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646183/ [7646184] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:21:19 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646184/ [7646185] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:27:18 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646185/ [7646186] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:14:56 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646186/ [7646187] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_monitoring_stack_basic} time: 00:27:52 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646187/ [7646188] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:09 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646188/ [7646191] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout} time: 00:14:39 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646191/ [7646192] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:15:05 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646192/ [7646193] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:35:07 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646193/ [7646194] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:22:16 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646194/ [7646195] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:15:54 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646195/ [7646196] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:29 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646196/ [7646197] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:06 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646197/ [7646199] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:20:20 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646199/ [7646201] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:18:42 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646201/ [7646203] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:18:42 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646203/ [7646205] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli} time: 00:19:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646205/ [7646206] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:24:23 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646206/ [7646208] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:18:50 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646208/ [7646209] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_ca_signed_key} time: 00:15:05 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646209/ [7646210] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:16:44 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646210/ [7646211] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:17:22 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646211/ [7646212] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:23:03 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646212/ [7646214] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:25:01 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646214/ [7646217] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon} time: 00:24:18 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646217/ [7646218] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} time: 00:14:28 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646218/ [7646219] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:03 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646219/ [7646220] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:13:07 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646220/ [7646222] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:52 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646222/ [7646224] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:14:25 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646224/ [7646225] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_cephadm_repos} time: 00:06:38 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646225/ [7646227] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:25:30 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646227/ [7646228] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:42 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646228/ [7646229] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_extra_daemon_features} time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646229/ [7646232] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption} time: 00:10:31 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646232/ [7646233] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:18:37 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646233/ [7646234] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:13:48 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646234/ [7646235] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:23:56 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646235/ [7646236] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:20:12 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646236/ [7646239] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:17 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646239/ [7646240] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:14:39 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646240/ [7646242] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:14:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646242/ [7646243] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646243/ [7646244] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:00 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646244/ [7646246] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout} time: 00:19:11 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646246/ [7646247] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:19:56 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646247/ [7646249] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:14:53 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646249/ [7646252] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_monitoring_stack_basic} time: 00:19:24 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646252/ [7646253] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:16:17 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646253/ [7646254] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:02 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646254/ [7646255] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:21:33 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646255/ [7646256] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646256/ [7646258] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_rgw_multisite} time: 00:20:33 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646258/ [7646265] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:34:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646265/ [7646266] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} time: 00:13:29 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646266/ [7646267] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} time: 00:12:20 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646267/ [7646269] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:05 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646269/ [7646270] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:34 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646270/ [7646271] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:11:45 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646271/ [7646272] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:16:38 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646272/ [7646273] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:40 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646273/ [7646274] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:17 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646274/ [7646276] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:21:52 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646276/ [7646279] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:13:48 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646279/ [7646281] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:33 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646281/ [7646283] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} time: 00:24:36 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646283/ [7646284] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:26 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646284/ [7646285] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:37 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646285/ [7646286] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:20:33 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646286/ [7646287] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:25:18 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646287/ [7646290] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:22 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646290/ [7646292] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} time: 00:24:28 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646292/ [7646293] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:55 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646293/ [7646295] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:50 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646295/ [7646296] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} time: 00:12:19 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646296/ [7646297] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646297/ [7646298] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:18:36 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646298/ [7646299] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:13:52 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646299/ [7646300] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:14:43 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646300/ [7646304] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:22:13 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646304/ [7646305] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:32 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646305/ [7646307] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_host_drain} time: 00:16:15 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646307/ [7646309] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} time: 00:14:13 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646309/ [7646310] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} time: 00:12:48 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646310/ [7646311] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:30 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646311/ [7646312] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:49 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646312/ [7646314] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:15:39 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646314/ [7646315] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:53 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646315/ [7646316] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:30 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646316/ [7646317] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646317/ [7646320] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:15:40 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646320/ [7646321] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:20:49 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646321/ [7646322] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:23 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646322/ [7646323] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} time: 00:14:08 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646323/ [7646324] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:22 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646324/ [7646326] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:14 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646326/ [7646327] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:24:04 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646327/ [7646329] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:45 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646329/ [7646330] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:14:12 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646330/ [7646331] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:14:58 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646331/ [7646332] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:11:49 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646332/ [7646334] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:14:37 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646334/ [7646335] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:21:59 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646335/ [7646337] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} time: 00:37:07 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646337/ [7646338] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:34 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646338/ [7646339] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:20:54 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646339/ [7646340] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:14:45 info: https://pulpito.ceph.com/teuthology-2024-04-08_21:08:03-orch-squid-distro-default-smithi/7646340/