Found 173 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654635/teuthology.log Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654635/teuthology.log Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Found 11 distinct failure reasons Assertion: 2024-04-13T00:56:47.820 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 13 00:56:47 smithi043 ceph-2fcd18ba-f92d-11ee-bc8d-c7b262605968-osd-1[237909]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x56060e1c4a57] 1 jobs: ['7654686'] suites: ['1-start', '2-thrash', '3-tasks/snaps-few-objects', 'fixed-2', 'msgr/async-v1only', 'orch/cephadm/thrash/{0-distro/centos_9.stream_runc', 'root}'] Failure: Command failed on smithi031 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7007fc74-f91e-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch ps'" 3 jobs: ['7654616', '7654695', '7654574'] suites intersection: [] suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon_election/classic}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.1}'] Failure: "2024-04-12T23:32:16.757661+0000 mon.a (mon.0) 329 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log 37 jobs suites intersection: [] suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '2-nfs', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/jaeger', '2-services/nfs', '2-services/nfs-ingress-rgw-bucket', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/yes', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch/cephadm/osds/{0-distro/centos_9.stream', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/centos_9.stream', 'orch/cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/ubuntu_22.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch/cephadm/with-work/{0-distro/centos_9.stream', 'orch/cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch/cephadm/with-work/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'workloads/cephadm_iscsi}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=98423fadeecbba811088c3136f8295dae6dfe870 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 3 jobs: ['7654557', '7654619', '7654687'] suites intersection: ['task/test_cephadm}'] suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_9.stream', 'orch/cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] Failure: reached maximum tries (51) after waiting for 300 seconds 5 jobs: ['7654539', '7654675', '7654602', '7654630', '7654531'] suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}'] Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 3 jobs: ['7654587', '7654647', '7654560'] suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}'] Failure: Command failed on smithi092 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 4 jobs: ['7654582', '7654624', '7654540', '7654666'] suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_22.04'] suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_22.04', 'rook/1.7.2}', 'rook/master}'] Dead: None 1 jobs: ['7654635'] Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=98423fadeecbba811088c3136f8295dae6dfe870 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 1 jobs: ['7654563'] suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] Failure: Command failed on smithi006 with status 5: 'sudo systemctl stop ceph-a29b4344-f91e-11ee-bc8d-c7b262605968@mon.a' 1 jobs: ['7654612'] suites: ['agent/off', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] Dead: b'2024-04-13T09:38:34.334 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7654547'] suites: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/classic', 'orch/cephadm/smoke/{0-distro/centos_9.stream', 'start}']