Found 89 jobs
Missing teuthology log /home/teuthworker/archive/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645905/teuthology.log
Found 7 distinct failure reasons
Failure: "2024-04-08T20:29:25.593601+0000 mon.smithi092 (mon.0) 598 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
63 jobs
suites intersection: []
suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '2-nfs', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/iscsi', '2-services/jaeger', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/no', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smb/{0-distro/centos_9.stream', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'roles', 'root}', 'start', 'start}', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'task/test_cephadm_timeout}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/deploy_smb_basic}', 'tasks/deploy_smb_domain}', 'tasks/deploy_smb_mgr_domain}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}']

Failure: reached maximum tries (51) after waiting for 300 seconds
5 jobs: ['7645927', '7645907', '7645963', '7645884', '7645956']
suites intersection: []
suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/deploy_smb_mgr_res_dom}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}']

Dead: None
2 jobs: ['7645905', '7645904']

Dead: b'2024-04-09T08:41:39.141 DEBUG:teuthology.exit:Finished running handlers'
3 jobs: ['7645959', '7645886', '7645894']
suites intersection: ['0-nvme-loop']
suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final}', 'agent/off', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'start}']

Failure: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4432bbac-f5ea-11ee-b649-cb9ed24678a4 -e sha1=fb6f9800392d73c10dc835c8e8e86c5884686270 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''
1 jobs: ['7645971']
suites: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
1 jobs: ['7645960']
suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']

Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
1 jobs: ['7645901']
suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}']