Found 95 jobs
Missing teuthology log /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581171/teuthology.log
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Found 12 distinct failure reasons
Failure: "2024-03-04T15:41:41.931377+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log
14 jobs: ['7581221', '7581253', '7581185', '7581178', '7581216', '7581161', '7581240', '7581238', '7581170', '7581184', '7581201', '7581231', '7581239', '7581218']
suites intersection: []
suites union: ['0-nvme-loop', '1-start', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}']

Failure: reached maximum tries (51) after waiting for 300 seconds
4 jobs: ['7581235', '7581163', '7581229', '7581188']
suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}']

Dead: None
6 jobs: ['7581171', '7581200', '7581172', '7581199', '7581198', '7581179']

Failure: Failed to reconnect to smithi073
3 jobs: ['7581207', '7581219', '7581174']
suites intersection: []
suites union: ['0-nvme-loop', '1-start', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress2', '3-final}', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'start', 'tasks/rotate-keys}']

Failure: "1709566765.979432 mon.smithi012 (mon.0) 330 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
1 jobs: ['7581196']
suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']

Failure: Command failed on smithi103 with status 1: 'sudo yum install -y kernel'
2 jobs: ['7581173', '7581206']
suites intersection: []
suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'task/test_rgw_multisite}', 'tasks/{0-from/reef/{v18.2.0}']

Failure: failed to install new kernel version within timeout
1 jobs: ['7581209']
suites: ['mon_election/classic', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'task/test_orch_cli}']

Failure: ['Failed to manage policy for boolean nagios_run_sudo: [Errno 11] Resource temporarily unavailable']
1 jobs: ['7581222']
suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_monitoring_stack_basic}']

Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
1 jobs: ['7581250']
suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}']

Failure: SSH connection to smithi165 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'
1 jobs: ['7581220']
suites: ['1-start', '2-thrash', '3-tasks/rados_api_tests', 'fixed-2', 'msgr/async', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'root}']

Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
1 jobs: ['7581246']
suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi090 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2 jobs: ['7581190', '7581244']
suites intersection: ['task/test_cephadm}']
suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']