Found 428 jobs
Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564941/teuthology.log
Found 12 distinct failure reasons
Failure: "2024-02-18T01:00:00.000244+0000 mon.smithi049 (mon.0) 326 : cluster [WRN]     osd.3 (root=default,host=smithi203) is down" in cluster log
28 jobs: ['7564996', '7565135', '7565104', '7565106', '7564801', '7565141', '7564822', '7564825', '7565021', '7565170', '7564978', '7565017', '7564874', '7564842', '7564952', '7564911', '7564893', '7565163', '7564868', '7565037', '7564776', '7565043', '7565088', '7564866', '7564880', '7564745', '7565039', '7564992']
suites intersection: []
suites union: ['0-nvme-loop', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-install/mimic', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/yes', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'backoff/peering_and_degraded', 'centos_8.stream_container_tools', 'ceph', 'clusters/{openstack', 'conf/{client', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados', 'rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/dashboard/{0-distro/ignorelist_health', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/with-work/{0-distro/ubuntu_18.04', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'roles', 'start', 'task/test_e2e}', 'task/test_nfs}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/{0-from/v16.2.4', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}']

Dead: None
3 jobs: ['7564941', '7564826', '7565147']

Failure: reached maximum tries (301) after waiting for 300 seconds
4 jobs: ['7565156', '7565167', '7565044', '7565122']
suites intersection: ['0-nvme-loop', '1-start', '3-final}']
suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress2', '2-services/rgw-ingress', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04']

Failure: expected string or bytes-like object
12 jobs: ['7565101', '7564773', '7564985', '7565134', '7564851', '7565168', '7564957', '7565064', '7564820', '7564884', '7565034', '7564918']
suites intersection: ['1-start', '2-thrash', 'fixed-2', 'rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}']
suites union: ['1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'fixed-2', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}']

Failure: Command failed on smithi100 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.100:/foouser /mnt/foo'"
2 jobs: ['7564793', '7564972']
suites intersection: []
suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']

Failure: Failed to reconnect to smithi138
5 jobs: ['7564936', '7565145', '7564858', '7564973', '7565093']
suites intersection: []
suites union: ['0-nvme-loop', '1-install/mimic-v1only', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-services/nfs2', '3-final}', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'backoff/peering_and_degraded', 'centos_8.stream_container_tools', 'ceph', 'clusters/{openstack', 'conf/{client', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'msgr-failures/osd-delay', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/osds/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'roles', 'tasks/{0-from/v16.2.4', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}']

Failure: Command failed on smithi189 with status 5: 'sudo systemctl stop ceph-e60c1720-ce03-11ee-95ba-87774f69a715@mon.smithi189'
6 jobs: ['7565157', '7565080', '7565056', '7565127', '7565110', '7565094']
suites intersection: ['3-upgrade-with-workload']
suites union: ['1-bootstrap/16.2.5', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'roles', 'tasks/{0-from/v16.2.4']

Failure: Command failed on smithi121 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4ab5f4ea-cdfd-11ee-95ba-87774f69a715 -e sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\''
2 jobs: ['7565083', '7564805']
suites intersection: []
suites union: ['1-bootstrap/16.2.4', '1-start', '2-nfs', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'mon_election/connectivity}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04']

Dead: b'2024-02-18T10:16:27.329 DEBUG:teuthology.exit:Finished running handlers'
1 jobs: ['7564747']
suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{ubuntu_latest}}']

Failure: ['Failed to update apt cache: unknown reason']
1 jobs: ['7564890']
suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04', 'start', 'tasks/rados_api_tests}']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81bd20d634209c7cb82c18be12b4b5a05643ebf1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2 jobs: ['7565152', '7565047']
suites intersection: ['rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}']

Failure: "2024-02-17T22:47:11.734736+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log
1 jobs: ['7564766']
suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}']