2024-03-11T20:10:37.314 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-11T20:10:37.314 INFO:teuthology.results:subset: '10/64' 2024-03-11T20:10:37.314 INFO:teuthology.results:seed: '6815' 2024-03-12T03:13:25.721 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-12T03:13:25.755 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-12T15:07:24.946 INFO:teuthology.results:Tests finished! gathering results... 2024-03-12T15:07:40.007 INFO:scrape:Found 173 jobs 2024-03-12T15:07:41.598 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:41.599 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:41.600 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:41.630 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:41.630 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:41.630 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:42.464 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:42.464 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:42.464 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:43.009 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:43.010 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:43.010 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.219 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.220 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.220 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.295 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.295 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:47.295 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.451 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.451 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.451 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.451 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.452 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.452 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.452 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.452 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.810 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.810 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.811 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.811 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.811 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.811 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.811 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:50.812 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.486 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.486 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.487 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.524 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.524 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:55.525 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:07:56.459 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591839/teuthology.log 2024-03-12T15:08:01.302 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.303 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.303 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.372 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.373 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.373 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.443 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.443 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.443 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.512 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.512 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.512 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.584 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.584 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:01.584 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.382 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.383 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.383 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.545 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.545 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.545 WARNING:scrape:Saw end of BT but not start 2024-03-12T15:08:03.852 INFO:scrape:Found 10 distinct failure reasons 2024-03-12T15:08:03.853 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-03-12T15:08:03.853 INFO:scrape:3 jobs: ['7591858', '7591725', '7591791'] 2024-03-12T15:08:03.853 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2024-03-12T15:08:03.853 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_9.stream', 'orch/cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-03-12T15:08:03.854 INFO:scrape: 2024-03-12T15:08:03.854 INFO:scrape:Failure: "2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log 2024-03-12T15:08:03.854 INFO:scrape:35 jobs 2024-03-12T15:08:03.855 INFO:scrape:suites intersection: [] 2024-03-12T15:08:03.855 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/jaeger', '2-services/nfs-keepalive-only', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/yes', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/centos_9.stream', 'orch/cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/ubuntu_22.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/with-work/{0-distro/centos_9.stream', 'orch/cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch/cephadm/with-work/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/reef/{reef}', 'workloads/cephadm_iscsi}'] 2024-03-12T15:08:03.855 INFO:scrape: 2024-03-12T15:08:03.855 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-03-12T15:08:03.855 INFO:scrape:1 jobs: ['7591798'] 2024-03-12T15:08:03.855 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-03-12T15:08:03.855 INFO:scrape: 2024-03-12T15:08:03.856 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-12T15:08:03.856 INFO:scrape:6 jobs: ['7591838', '7591806', '7591877', '7591848', '7591751', '7591776'] 2024-03-12T15:08:03.856 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-12T15:08:03.856 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-12T15:08:03.856 INFO:scrape: 2024-03-12T15:08:03.856 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-12T15:08:03.856 INFO:scrape:1 jobs: ['7591892'] 2024-03-12T15:08:03.856 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-12T15:08:03.856 INFO:scrape: 2024-03-12T15:08:03.857 INFO:scrape:Failure: Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-12T15:08:03.857 INFO:scrape:2 jobs: ['7591862', '7591741'] 2024-03-12T15:08:03.857 INFO:scrape:suites intersection: [] 2024-03-12T15:08:03.857 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-repo_digest/defaut', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-12T15:08:03.857 INFO:scrape: 2024-03-12T15:08:03.857 INFO:scrape:Failure: Command failed on smithi003 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 2024-03-12T15:08:03.857 INFO:scrape:4 jobs: ['7591802', '7591886', '7591844', '7591760'] 2024-03-12T15:08:03.857 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_22.04'] 2024-03-12T15:08:03.858 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_22.04', 'rook/1.7.2}', 'rook/master}'] 2024-03-12T15:08:03.858 INFO:scrape: 2024-03-12T15:08:03.858 INFO:scrape:Dead: b'2024-03-12T14:47:08.109 DEBUG:teuthology.exit:Finished running handlers' 2024-03-12T15:08:03.858 INFO:scrape:3 jobs: ['7591840', '7591867', '7591847'] 2024-03-12T15:08:03.858 INFO:scrape:suites intersection: ['0-nvme-loop'] 2024-03-12T15:08:03.858 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs2', '2-services/rgw', '3-final}', 'agent/on', 'fixed-2', 'mon_election/classic', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'start}'] 2024-03-12T15:08:03.858 INFO:scrape: 2024-03-12T15:08:03.858 INFO:scrape:Dead: None 2024-03-12T15:08:03.858 INFO:scrape:1 jobs: ['7591839'] 2024-03-12T15:08:03.858 INFO:scrape: 2024-03-12T15:08:03.859 INFO:scrape:Failure: Command failed on smithi077 with status 5: 'sudo systemctl stop ceph-f9e0f334-e01d-11ee-95c8-87774f69a715@mon.a' 2024-03-12T15:08:03.859 INFO:scrape:1 jobs: ['7591855'] 2024-03-12T15:08:03.859 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-03-12T15:08:03.859 INFO:scrape: 2024-03-12T15:08:03.865 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/ failed: 53 dead: 4 running: 0 waiting: 0 queued: 0 passed: 116 Fail ================================================================= [7591725] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:17:19 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591725/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4692255d0464f8ab7c0a019890ae068 Command failed (workunit test cephadm/test_cephadm.sh) on smithi096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7591726] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:51:45 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591726/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591726/ "2024-03-11T20:40:00.000210+0000 mon.a (mon.0) 1881 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7591731] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:35:24 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591731/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591731/ "2024-03-11T20:50:00.000265+0000 mon.smithi063 (mon.0) 488 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 38/228 objects degraded (16.667%), 15 pgs degraded" in cluster log [7591732] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:47 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591732/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591732/ "2024-03-11T20:33:38.064181+0000 mon.smithi083 (mon.0) 497 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7591737] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:25:27 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591737/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591737/ "2024-03-11T20:29:56.950776+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591740] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591740/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591740/ "2024-03-11T20:29:49.739247+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591741] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:43:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591741/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4252657b227b4a87b3b954b1933583cc Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 05bd216a-dfe6-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7591745] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:59:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/ "2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7591749] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:51:48 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/ "2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7591751] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:36 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591751/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591751/ reached maximum tries (51) after waiting for 300 seconds [7591759] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:30:23 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591759/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591759/ "2024-03-11T21:00:00.000141+0000 mon.a (mon.0) 1565 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7591760] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:07:30 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591760/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=311c6f22863c4e4d870344e235e0cbcf Command failed on smithi052 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7591761] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:38:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591761/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591761/ "2024-03-11T20:47:42.583795+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7591767] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:31:58 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591767/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591767/ "2024-03-11T20:56:41.184995+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7591773] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:36:17 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591773/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591773/ "2024-03-11T21:10:00.000160+0000 mon.a (mon.0) 875 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7591776] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591776/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591776/ reached maximum tries (51) after waiting for 300 seconds [7591781] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:36 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591781/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591781/ "2024-03-11T21:01:49.482049+0000 mon.a (mon.0) 245 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591785] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:28:55 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591785/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591785/ "2024-03-11T21:06:29.207460+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7591789] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:50:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591789/ "2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7591791] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:13:17 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591791/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5222bbc1a5c34038a73fee57ce121de6 Command failed (workunit test cephadm/test_cephadm.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7591797] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:52:57 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591797/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591797/ "2024-03-11T21:30:00.000181+0000 mon.a (mon.0) 2380 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled" in cluster log [7591798] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:30 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591798/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67542fd492cc4b2182bdd969c5bac1bb Command failed (workunit test cephadm/test_repos.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7591800] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:39 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/ "2024-03-11T21:17:08.454036+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591802] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:07:32 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591802/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591802/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed7b47a199244b2d8b04acffb2f4e732 Command failed on smithi003 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7591804] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:56:11 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/ "2024-03-11T21:30:00.002162+0000 mon.a (mon.0) 921 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7591806] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591806/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591806/ reached maximum tries (51) after waiting for 300 seconds [7591810] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 01:03:21 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591810/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591810/ "2024-03-11T21:26:05.260760+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7591815] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:36:48 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591815/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591815/ "2024-03-11T21:33:01.944154+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7591817] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:30:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591817/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591817/ "2024-03-12T03:00:00.000133+0000 mon.a (mon.0) 1750 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7591820] orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:12 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591820/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591820/ "2024-03-12T02:53:52.503774+0000 mon.smithi003 (mon.0) 491 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7591827] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:59 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/ "2024-03-12T02:52:23.500567+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591831] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591831/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591831/ "2024-03-12T03:00:00.000207+0000 mon.smithi106 (mon.0) 725 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log [7591835] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:33:24 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591835/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591835/ "2024-03-12T02:58:40.818683+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7591836] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:49:24 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591836/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591836/ "2024-03-12T03:00:16.177735+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7591838] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:07 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591838/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591838/ reached maximum tries (51) after waiting for 300 seconds [7591844] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:07:30 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591844/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591844/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16dcbf97106c4d3a9bab5ea605b28b05 Command failed on smithi001 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7591846] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:12:13 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591846/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591846/ "2024-03-12T02:59:19.880343+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591848] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:37 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591848/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591848/ reached maximum tries (51) after waiting for 300 seconds [7591853] orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:04:50 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591853/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591853/ "2024-03-12T03:30:00.000296+0000 mon.a (mon.0) 2422 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log [7591854] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:52:20 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591854/ "2024-03-12T03:20:00.000255+0000 mon.a (mon.0) 2080 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log [7591855] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:09 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591855/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f45cb85c3f124a93a0f650f9a5c9cf89 Command failed on smithi077 with status 5: 'sudo systemctl stop ceph-f9e0f334-e01d-11ee-95c8-87774f69a715@mon.a' [7591858] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:15:18 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591858/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4581313842844f6a6f4c28b35274d23 Command failed (workunit test cephadm/test_cephadm.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7591859] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:56:34 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591859/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591859/ "2024-03-12T03:11:14.956338+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7591860] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:20:04 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591860/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591860/ "2024-03-12T03:20:00.000275+0000 mon.smithi083 (mon.0) 818 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)" in cluster log [7591862] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:10:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1161e9a464c4e0d82845819cd66a215 Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7591869] orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 01:07:55 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/ "2024-03-12T03:40:00.000170+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7591870] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591870/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591870/ "2024-03-12T03:30:00.000144+0000 mon.a (mon.0) 1190 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log [7591877] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:23 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591877/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591877/ reached maximum tries (51) after waiting for 300 seconds [7591883] orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:48:56 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/ "2024-03-12T03:57:51.392089+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7591884] orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:40:34 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591884/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591884/ "2024-03-12T03:35:06.045695+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7591886] orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:07:18 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591886/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591886/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8a9c4113f504c1485c7a43e8b9ca7d8 Command failed on smithi081 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7591888] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:16:40 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591888/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591888/ "2024-03-12T03:33:48.852760+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7591892] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:47:07 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591892/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591892/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e8d2d0c9eb04aeaa52ba04aa5bb7d4f Command failed (workunit test suites/fsstress.sh) on smithi115 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' Dead ================================================================= [7591839] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591839/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591839/ Error reimaging machines: Failed to power on smithi028 [7591840] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591840/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591840/ hit max job timeout [7591847] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591847/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591847/ hit max job timeout [7591867] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/ hit max job timeout Pass ================================================================= [7591724] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:39 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591724/ [7591727] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:18:57 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591727/ [7591728] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rotate-keys} time: 00:23:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591728/ [7591729] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:22:17 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591729/ [7591730] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:47 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591730/ [7591733] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:46:56 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591733/ [7591734] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption} time: 00:10:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591734/ [7591735] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} time: 00:12:05 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591735/ [7591736] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:14 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591736/ [7591738] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:13:13 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591738/ [7591739] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:31 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591739/ [7591742] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:14:36 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591742/ [7591743] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:18:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591743/ [7591744] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:20:19 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591744/ [7591746] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:25:19 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591746/ [7591747] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:14:41 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591747/ [7591748] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591748/ [7591750] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} time: 00:23:19 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591750/ [7591752] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout} time: 00:14:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591752/ [7591753] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:48 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591753/ [7591754] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:21:48 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591754/ [7591755] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591755/ [7591756] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:15:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591756/ [7591757] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:12:13 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591757/ [7591758] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:20:20 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591758/ [7591762] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:15:59 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591762/ [7591763] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:19:58 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591763/ [7591764] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591764/ [7591765] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli} time: 00:19:36 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591765/ [7591766] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:24:34 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591766/ [7591768] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:44 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591768/ [7591769] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:03 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591769/ [7591770] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:37 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591770/ [7591771] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:19:22 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591771/ [7591772] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:14:43 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591772/ [7591774] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:20:07 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591774/ [7591775] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:15:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591775/ [7591777] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon} time: 00:23:35 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591777/ [7591778] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} time: 00:11:31 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591778/ [7591779] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:18 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591779/ [7591780] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:46 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591780/ [7591782] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:15:14 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591782/ [7591783] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:15:11 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591783/ [7591784] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:23:57 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591784/ [7591786] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:11:20 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591786/ [7591787] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:26:38 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591787/ [7591788] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:55 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591788/ [7591790] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591790/ [7591792] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591792/ [7591793] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption} time: 00:10:31 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591793/ [7591794] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:18:29 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591794/ [7591795] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:14:29 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591795/ [7591796] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:21:04 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591796/ [7591799] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:14:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591799/ [7591801] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:17:49 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591801/ [7591803] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:16 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591803/ [7591805] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} time: 00:15:30 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591805/ [7591807] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout} time: 00:19:38 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591807/ [7591808] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:21:41 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591808/ [7591809] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:25:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591809/ [7591811] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:19:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591811/ [7591812] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} time: 00:17:43 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591812/ [7591813] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:58 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591813/ [7591814] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:16:40 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591814/ [7591816] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:22:04 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591816/ [7591818] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591818/ [7591819] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:40 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591819/ [7591821] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:36:24 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591821/ [7591822] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591822/ [7591823] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} time: 00:11:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591823/ [7591824] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:24:54 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591824/ [7591825] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:00 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591825/ [7591826] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:35 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591826/ [7591828] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:22 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591828/ [7591829] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:32 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591829/ [7591830] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:19:31 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591830/ [7591832] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591832/ [7591833] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:14:41 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591833/ [7591834] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:24:10 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591834/ [7591837] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite} time: 00:18:06 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591837/ [7591841] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:39 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591841/ [7591842] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:20:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591842/ [7591843] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:21:59 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591843/ [7591845] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591845/ [7591849] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} time: 00:12:09 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591849/ [7591850] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:12:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591850/ [7591851] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:54 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591851/ [7591852] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:11 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591852/ [7591856] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:16:01 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591856/ [7591857] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:14:31 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591857/ [7591861] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:25 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591861/ [7591863] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} time: 00:22:22 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591863/ [7591864] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} time: 00:16:21 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591864/ [7591865] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:17:26 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591865/ [7591866] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:13:41 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591866/ [7591868] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} time: 00:08:30 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591868/ [7591871] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:17:48 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591871/ [7591872] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:25 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591872/ [7591873] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591873/ [7591874] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:19:33 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591874/ [7591875] orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:17:47 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591875/ [7591876] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:45 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591876/ [7591878] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} time: 00:16:35 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591878/ [7591879] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:26:54 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591879/ [7591880] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} time: 00:25:10 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591880/ [7591881] orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:38:56 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591881/ [7591882] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:23:29 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591882/ [7591885] orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:16:19 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591885/ [7591887] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:16:54 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591887/ [7591889] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:16:02 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591889/ [7591890] orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:17:52 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591890/ [7591891] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:10 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591891/ [7591893] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} time: 00:37:14 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591893/ [7591894] orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:51 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591894/ [7591895] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:25:08 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591895/ [7591896] orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:21:57 info: https://pulpito.ceph.com/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591896/