2023-11-27T13:48:34.994 INFO:root:teuthology version: 0.0.1.dev205+g2442239
2023-11-27T13:48:34.994 INFO:teuthology.results:subset: '1/14'
2023-11-27T13:48:34.994 INFO:teuthology.results:seed: '8632'
2023-11-27T14:41:42.839 INFO:root:teuthology version: 0.0.1.dev205+g2442239
2023-11-27T14:41:42.867 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-27T16:20:50.347 INFO:teuthology.results:Tests finished! gathering results...
2023-11-27T16:20:56.868 INFO:scrape:Found 97 jobs
2023-11-27T16:21:05.407 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.408 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.408 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.438 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.438 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.439 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.464 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.464 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:05.464 WARNING:scrape:Saw end of BT but not start
2023-11-27T16:21:10.020 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467739/teuthology.log
2023-11-27T16:21:20.999 INFO:scrape:Found 14 distinct failure reasons
2023-11-27T16:21:20.999 INFO:scrape:Failure: "/var/log/ceph/20c6976a-8d30-11ee-95a2-87774f69a715/ceph-mon.smithi090.log:2023-11-27T14:23:26.024+0000 7fce98a7d700 10 mon.smithi090@0(leader).log v85  logging 2023-11-27T14:23:25.190106+0000 mgr.smithi090.plvvgq (mgr.14209) 15 : cephadm [ERR] Failed while placing ceph-exporter.smithi203 on smithi203: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-20c6976a-8d30-11ee-95a2-87774f69a715-ceph-exporter-smithi203" in cluster log
2023-11-27T16:21:20.999 INFO:scrape:30 jobs
2023-11-27T16:21:21.000 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', 'overrides/ignorelist}']
2023-11-27T16:21:21.000 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all', '2-ops/rm-zap-add', '2-ops/rm-zap-flag', '2-ops/rm-zap-wait', '2-ops/rmdir-reactivate', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '3-final', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'overrides/ignorelist}']
2023-11-27T16:21:21.000 INFO:scrape:
2023-11-27T16:21:21.001 INFO:scrape:Failure: Command failed on smithi017 with status 1: 'sudo yum -y install ceph'
2023-11-27T16:21:21.001 INFO:scrape:11 jobs: ['7467738', '7467755', '7467781', '7467789', '7467768', '7467702', '7467746', '7467752', '7467793', '7467735', '7467745']
2023-11-27T16:21:21.001 INFO:scrape:suites intersection: []
2023-11-27T16:21:21.001 INFO:scrape:suites union: ['1-start', '2-node-mgr', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orchestrator_cli', 'overrides/ignorelist', 'overrides/ignorelist}', 'root}', 'start', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'tasks/rados_api_tests}', 'tasks/rotate-keys}']
2023-11-27T16:21:21.002 INFO:scrape:
2023-11-27T16:21:21.002 INFO:scrape:Failure: "/var/log/ceph/e5bce716-8d32-11ee-95a2-87774f69a715/ceph-mon.a.log:2023-11-27T14:43:32.890+0000 7f6041eff700  0 log_channel(cluster) log [WRN] : Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
2023-11-27T16:21:21.002 INFO:scrape:25 jobs: ['7467754', '7467762', '7467722', '7467787', '7467788', '7467713', '7467717', '7467743', '7467783', '7467779', '7467766', '7467708', '7467718', '7467730', '7467737', '7467728', '7467729', '7467715', '7467794', '7467777', '7467758', '7467734', '7467723', '7467791', '7467724']
2023-11-27T16:21:21.002 INFO:scrape:suites intersection: []
2023-11-27T16:21:21.003 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-thrash', '3-inline/yes', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'osd}', 'overrides/ignorelist', 'overrides/ignorelist}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy']
2023-11-27T16:21:21.003 INFO:scrape:
2023-11-27T16:21:21.003 INFO:scrape:Failure: "/var/log/ceph/0392cb28-8d35-11ee-95a2-87774f69a715/ceph-mon.a.log:2023-11-27T15:09:00.497+0000 7f7e6f58d700 10 mon.a@0(leader).log v533  logging 2023-11-27T15:08:59.879469+0000 mgr.x (mgr.24466) 28 : cephadm [ERR] Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2023-11-27T15:08:59.872+0000 7efd837fe700  1 -- 172.21.15.188:0/4189649617 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7efd74003770 con 0x7efd84108ca0" in cluster log
2023-11-27T16:21:21.003 INFO:scrape:1 jobs: ['7467761']
2023-11-27T16:21:21.004 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'overrides/ignorelist', 'start', 'tasks/rotate-keys}']
2023-11-27T16:21:21.004 INFO:scrape:
2023-11-27T16:21:21.004 INFO:scrape:Failure: "/var/log/ceph/1ca71b60-8d34-11ee-95a2-87774f69a715/ceph-mon.a.log:2023-11-27T15:02:29.415+0000 7f42e48ab700 10 mon.a@0(leader).log v446  logging 2023-11-27T15:02:29.265649+0000 mgr.a (mgr.14152) 463 : cephadm [ERR] host smithi184 `cephadm ceph-volume` failed: Cannot decode JSON: " in cluster log
2023-11-27T16:21:21.004 INFO:scrape:1 jobs: ['7467772']
2023-11-27T16:21:21.004 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'overrides/ignorelist', 'task/test_host_drain}']
2023-11-27T16:21:21.004 INFO:scrape:
2023-11-27T16:21:21.005 INFO:scrape:Failure: "/var/log/ceph/87f606b8-8d3b-11ee-95a2-87774f69a715/ceph-mon.smithi059.log:2023-11-27T15:47:54.920+0000 7f54129ef700  0 log_channel(cluster) log [WRN] : Replacing daemon mds.cephfs.smithi059.ejobnm as rank 0 with standby daemon mds.cephfs.smithi167.gwscqw" in cluster log
2023-11-27T16:21:21.005 INFO:scrape:3 jobs: ['7467775', '7467741', '7467726']
2023-11-27T16:21:21.005 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2023-11-27T16:21:21.005 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2023-11-27T16:21:21.006 INFO:scrape:
2023-11-27T16:21:21.006 INFO:scrape:Failure: "/var/log/ceph/97cd1f32-8d3a-11ee-95a2-87774f69a715/ceph-mon.a.log:2023-11-27T15:49:59.999+0000 7fbf16710700  0 log_channel(cluster) log [WRN] :     application not enabled on pool 'ceph_test_rados_api_asio'" in cluster log
2023-11-27T16:21:21.006 INFO:scrape:2 jobs: ['7467767', '7467778']
2023-11-27T16:21:21.006 INFO:scrape:suites intersection: ['fixed-2', 'overrides/ignorelist']
2023-11-27T16:21:21.007 INFO:scrape:suites union: ['1-start', '2-thrash', '3-tasks/snaps-few-objects', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'overrides/ignorelist', 'root}', 'start', 'tasks/rados_api_tests}']
2023-11-27T16:21:21.007 INFO:scrape:
2023-11-27T16:21:21.007 INFO:scrape:Failure: "/var/log/ceph/9a16a460-8d31-11ee-95a2-87774f69a715/ceph-mon.smithi002.log:2023-11-27T14:42:01.502+0000 7f40bbaa5700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 58/354 objects degraded (16.384%), 17 pgs degraded (PG_DEGRADED)" in cluster log
2023-11-27T16:21:21.007 INFO:scrape:2 jobs: ['7467711', '7467771']
2023-11-27T16:21:21.009 INFO:scrape:suites intersection: []
2023-11-27T16:21:21.009 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'osd}', 'overrides/ignorelist', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'start}', 'syntax}', 'tasks/{0-from/quincy']
2023-11-27T16:21:21.009 INFO:scrape:
2023-11-27T16:21:21.010 INFO:scrape:Failure: Command failed on smithi112 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest'
2023-11-27T16:21:21.010 INFO:scrape:2 jobs: ['7467710', '7467774']
2023-11-27T16:21:21.010 INFO:scrape:suites intersection: ['fixed-2', 'overrides/ignorelist', 'start', 'tasks/rados_python}']
2023-11-27T16:21:21.011 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'overrides/ignorelist', 'start', 'tasks/rados_python}']
2023-11-27T16:21:21.011 INFO:scrape:
2023-11-27T16:21:21.011 INFO:scrape:Dead: None
2023-11-27T16:21:21.011 INFO:scrape:2 jobs: ['7467739', '7467704']
2023-11-27T16:21:21.011 INFO:scrape:
2023-11-27T16:21:21.012 INFO:scrape:Failure: Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1877a870-8d2f-11ee-95a2-87774f69a715 -e sha1=a501ccc0f4e7124633d52671cd089579114504d3 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2023-11-27T16:21:21.012 INFO:scrape:1 jobs: ['7467703']
2023-11-27T16:21:21.012 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'overrides/ignorelist}']
2023-11-27T16:21:21.012 INFO:scrape:
2023-11-27T16:21:21.012 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a501ccc0f4e7124633d52671cd089579114504d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
2023-11-27T16:21:21.013 INFO:scrape:1 jobs: ['7467747']
2023-11-27T16:21:21.013 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'overrides/ignorelist', 'task/test_adoption}']
2023-11-27T16:21:21.013 INFO:scrape:
2023-11-27T16:21:21.013 INFO:scrape:Failure: "/var/log/ceph/be21eaf6-8d34-11ee-95a2-87774f69a715/ceph-mon.c.log:2023-11-27T14:56:14.188+0000 7ff8e7e4b700  7 mon.c@2(synchronizing).log v63 update_from_paxos applying incremental log 62 2023-11-27T14:56:12.206253+0000 mon.a (mon.0) 176 : cluster [WRN]     mon.c (rank 2) addr [v2:172.21.15.61:3301/0,v1:172.21.15.61:6790/0] is down (out of quorum)" in cluster log
2023-11-27T16:21:21.013 INFO:scrape:1 jobs: ['7467785']
2023-11-27T16:21:21.014 INFO:scrape:suites: ['0-nvme-loop', 'agent/off', 'fixed-2', 'mon_election/connectivity', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'overrides/ignorelist', 'start}']
2023-11-27T16:21:21.014 INFO:scrape:
2023-11-27T16:21:21.014 INFO:scrape:Failure: "/var/log/ceph/eddc4af8-8d2e-11ee-95a2-87774f69a715/ceph-mon.smithi007.log:2023-11-27T14:16:57.693+0000 7f3a42b84700 10 mon.smithi007@0(leader).log v105  logging 2023-11-27T14:16:57.043705+0000 mgr.smithi007.rgyxeq (mgr.14195) 17 : cephadm [ERR] Failed while placing ceph-exporter.smithi183 on smithi183: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-eddc4af8-8d2e-11ee-95a2-87774f69a715-ceph-exporter-smithi183" in cluster log
2023-11-27T16:21:21.014 INFO:scrape:1 jobs: ['7467712']
2023-11-27T16:21:21.015 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'overrides/ignorelist}']
2023-11-27T16:21:21.015 INFO:scrape:
2023-11-27T16:21:21.020 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/
failed:  81
dead:    2
running: 0
waiting: 0
queued:  0
passed:  14


Fail
=================================================================
[7467699]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:20:22
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467699/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467699/

    "/var/log/ceph/e526bd40-8d2d-11ee-95a2-87774f69a715/ceph-
    mon.smithi081.log:2023-11-27T14:08:00.597+0000 7fb881002700 10
    mon.smithi081@0(leader).log v103  logging 2023-11-27T14:07:59.834111+0000
    mgr.smithi081.kjmfrx (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi193 on smithi193: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-e526bd40-8d2d-11ee-95a2-87774f69a715-ceph-
    exporter-smithi193" in cluster log

[7467700]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:46
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467700/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467700/

    "/var/log/ceph/ea89cbaa-8d2e-11ee-95a2-87774f69a715/ceph-
    mon.smithi050.log:2023-11-27T14:14:59.874+0000 7fc473020700 10
    mon.smithi050@0(leader).log v72  logging 2023-11-27T14:14:59.786066+0000
    mgr.smithi050.xhtbuh (mgr.14213) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi164 on smithi164: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-ea89cbaa-8d2e-11ee-95a2-87774f69a715-ceph-
    exporter-smithi164" in cluster log

[7467702]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity overrides/ignorelist task/test_host_drain}
-----------------------------------------------------------------
time:   00:11:39
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467702/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467702/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06ccaafb07754eb887f0b87481737898

    Command failed on smithi100 with status 1: 'sudo yum -y install ceph'

[7467703]  orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:36:04
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467703/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467703/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da39c808295f45c8961b002e15077801

    Command failed on smithi019 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1877a870-8d2f-11ee-95a2-87774f69a715 -e
    sha1=a501ccc0f4e7124633d52671cd089579114504d3 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7467708]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:21:19
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467708/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467708/

    "/var/log/ceph/5de6171e-8d31-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:32:06.093+0000 7f83287aa700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum)" in
    cluster log

[7467709]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:54
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467709/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467709/

    "/var/log/ceph/ea4faace-8d2e-11ee-95a2-87774f69a715/ceph-
    mon.smithi027.log:2023-11-27T14:15:26.134+0000 7f2c2df3a700 10
    mon.smithi027@0(leader).log v84  logging 2023-11-27T14:15:26.084379+0000
    mgr.smithi027.lqsuyn (mgr.14211) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi153 on smithi153: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-ea4faace-8d2e-11ee-95a2-87774f69a715-ceph-
    exporter-smithi153" in cluster log

[7467710]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only overrides/ignorelist start tasks/rados_python}
-----------------------------------------------------------------
time:   00:09:41
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467710/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467710/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c88a477256924cf5914ddd706ba69cff

    Command failed on smithi112 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest
    python3-pytest'

[7467711]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:40:22
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467711/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467711/

    "/var/log/ceph/9a16a460-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi002.log:2023-11-27T14:42:01.502+0000 7f40bbaa5700  0
    log_channel(cluster) log [WRN] : Health check failed: Degraded data
    redundancy: 58/354 objects degraded (16.384%), 17 pgs degraded
    (PG_DEGRADED)" in cluster log

[7467712]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:24:00
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467712/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467712/

    "/var/log/ceph/eddc4af8-8d2e-11ee-95a2-87774f69a715/ceph-
    mon.smithi007.log:2023-11-27T14:16:57.693+0000 7f3a42b84700 10
    mon.smithi007@0(leader).log v105  logging 2023-11-27T14:16:57.043705+0000
    mgr.smithi007.rgyxeq (mgr.14195) 17 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi183 on smithi183: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-eddc4af8-8d2e-11ee-95a2-87774f69a715-ceph-
    exporter-smithi183" in cluster log

[7467713]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only overrides/ignorelist root}
-----------------------------------------------------------------
time:   01:05:34
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467713/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467713/

    "/var/log/ceph/8d0fb33a-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:32:41.786+0000 7f7b731d0700  0 log_channel(cluster)
    log [WRN] : Health check failed: Degraded data redundancy: 72/594 objects
    degraded (12.121%), 17 pgs degraded (PG_DEGRADED)" in cluster log

[7467715]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async overrides/ignorelist start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:38:40
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467715/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467715/

    "/var/log/ceph/576e3f08-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:18:39.692+0000 7f8cb4655700  0 log_channel(cluster)
    log [WRN] :     mon.c (rank 2) addr
    [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] is down (out of quorum)"
    in cluster log

[7467716]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:22:44
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467716/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467716/

    "/var/log/ceph/34ac2bf4-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi098.log:2023-11-27T14:32:22.706+0000 7f26e98b5700 10
    mon.smithi098@0(leader).log v135  logging 2023-11-27T14:32:21.774477+0000
    mgr.smithi098.nepfah (mgr.14265) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi113 on smithi113: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-34ac2bf4-8d31-11ee-95a2-87774f69a715-ceph-
    exporter-smithi113" in cluster log

[7467717]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity overrides/ignorelist}
-----------------------------------------------------------------
time:   00:41:56
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467717/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467717/

    "/var/log/ceph/7ecaa46a-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:18:54.432+0000 7f9e2f157700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum)"
    in cluster log

[7467718]  orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity overrides/ignorelist task/test_monitoring_stack_basic}
-----------------------------------------------------------------
time:   00:25:31
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467718/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467718/

    "/var/log/ceph/bd8f953e-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:22:38.661+0000 7fb8582d2700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] is down (out of quorum)"
    in cluster log

[7467719]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:49
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467719/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467719/

    "/var/log/ceph/71a5c4fe-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.smithi040.log:2023-11-27T14:19:36.368+0000 7fa6a9535700 10
    mon.smithi040@0(leader).log v94  logging 2023-11-27T14:19:36.032985+0000
    mgr.smithi040.uumxiq (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi119 on smithi119: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-71a5c4fe-8d2f-11ee-95a2-87774f69a715-ceph-
    exporter-smithi119" in cluster log

[7467721]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:22:05
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467721/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467721/

    "/var/log/ceph/4c633b3e-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi012.log:2023-11-27T14:33:23.318+0000 7fe66cfb9700 10
    mon.smithi012@0(leader).log v135  logging 2023-11-27T14:33:22.837067+0000
    mgr.smithi012.jbefcm (mgr.14271) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi156 on smithi156: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-4c633b3e-8d31-11ee-95a2-87774f69a715-ceph-
    exporter-smithi156" in cluster log

[7467722]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only overrides/ignorelist root}
-----------------------------------------------------------------
time:   00:37:13
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467722/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467722/

    "/var/log/ceph/16ede02a-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:38:54.883+0000 7f1cac43d700  0 log_channel(cluster)
    log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects
    degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

[7467723]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async overrides/ignorelist start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:43:23
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467723/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467723/

    "/var/log/ceph/043b9206-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:49:59.999+0000 7ff30c600700  0 log_channel(cluster)
    log [WRN] :     application not enabled on pool 'ceph_test_rados_api_asio'"
    in cluster log

[7467724]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic overrides/ignorelist task/test_orch_cli}
-----------------------------------------------------------------
time:   00:21:32
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467724/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467724/

    "/var/log/ceph/35500e08-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:39:50.385+0000 7efdb493e700  0 log_channel(cluster)
    log [WRN] : Health check failed: cephadm background work is paused
    (CEPHADM_PAUSED)" in cluster log

[7467725]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:20:01
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467725/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467725/

    "/var/log/ceph/c54c85e8-8d2f-11ee-95a2-87774f69a715/ceph-
    mon.smithi088.log:2023-11-27T14:21:34.999+0000 7f5b0b5e0700 10
    mon.smithi088@0(leader).log v100  logging 2023-11-27T14:21:34.681321+0000
    mgr.smithi088.hpjvvs (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi092 on smithi092: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-c54c85e8-8d2f-11ee-95a2-87774f69a715-ceph-
    exporter-smithi092" in cluster log

[7467726]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:36:05
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467726/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467726/

    "/var/log/ceph/e98f0474-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi026.log:2023-11-27T14:39:15.038+0000 7f502acd5700  0
    log_channel(cluster) log [WRN] : Replacing daemon
    mds.cephfs.smithi026.xnducb as rank 0 with standby daemon
    mds.cephfs.smithi204.eumnht" in cluster log

[7467727]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:16:44
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467727/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467727/

    "/var/log/ceph/20c6976a-8d30-11ee-95a2-87774f69a715/ceph-
    mon.smithi090.log:2023-11-27T14:23:26.024+0000 7fce98a7d700 10
    mon.smithi090@0(leader).log v85  logging 2023-11-27T14:23:25.190106+0000
    mgr.smithi090.plvvgq (mgr.14209) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi203 on smithi203: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-20c6976a-8d30-11ee-95a2-87774f69a715-ceph-
    exporter-smithi203" in cluster log

[7467728]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity overrides/ignorelist task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:25:11
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467728/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467728/

    "/var/log/ceph/7c0ffe0c-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:40:26.726+0000 7f00adf66700  0 log_channel(cluster)
    log [WRN] :     mon.d (rank 3) addr
    [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum)"
    in cluster log

[7467729]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only overrides/ignorelist start tasks/rados_python}
-----------------------------------------------------------------
time:   00:24:54
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467729/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467729/

    "/var/log/ceph/ce91c7ce-8d31-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:34:45.130+0000 7ff74f15c700  0 log_channel(cluster)
    log [WRN] :     mon.c (rank 2) addr v1:172.21.15.83:6790/0 is down (out of
    quorum)" in cluster log

[7467730]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:14:21
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467730/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467730/

    "/var/log/ceph/c1625032-8d31-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:34:47.904+0000 7f3efb518700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum)"
    in cluster log

[7467731]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate overrides/ignorelist}
-----------------------------------------------------------------
time:   00:16:48
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467731/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467731/

    "/var/log/ceph/395af122-8d30-11ee-95a2-87774f69a715/ceph-
    mon.smithi133.log:2023-11-27T14:24:00.845+0000 7fc2fc06d700 10
    mon.smithi133@0(leader).log v79  logging 2023-11-27T14:23:59.893962+0000
    mgr.smithi133.qwzcwd (mgr.14207) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi142 on smithi142: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-395af122-8d30-11ee-95a2-87774f69a715-ceph-
    exporter-smithi142" in cluster log

[7467733]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:18
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467733/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467733/

    "/var/log/ceph/8c3cd068-8d30-11ee-95a2-87774f69a715/ceph-
    mon.smithi061.log:2023-11-27T14:27:45.899+0000 7f2334e6b700 10
    mon.smithi061@0(leader).log v106  logging 2023-11-27T14:27:45.617571+0000
    mgr.smithi061.cdovqy (mgr.14195) 17 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi170 on smithi170: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-8c3cd068-8d30-11ee-95a2-87774f69a715-ceph-
    exporter-smithi170" in cluster log

[7467734]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async overrides/ignorelist root}
-----------------------------------------------------------------
time:   00:38:04
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467734/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467734/

    "/var/log/ceph/0af2ce70-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:39:59.999+0000 7f7a9645b700  0 log_channel(cluster)
    log [WRN] : [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in
    cluster log

[7467735]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic overrides/ignorelist task/test_rgw_multisite}
-----------------------------------------------------------------
time:   00:11:23
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467735/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467735/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c57f88b8dba0423d8182606ce8e91d4e

    Command failed on smithi071 with status 1: 'sudo yum -y install ceph'

[7467736]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:24
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467736/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467736/

    "/var/log/ceph/922469d0-8d32-11ee-95a2-87774f69a715/ceph-
    mon.smithi046.log:2023-11-27T14:40:45.905+0000 7f5f70ca9700 10
    mon.smithi046@0(leader).log v81  logging 2023-11-27T14:40:45.139289+0000
    mgr.smithi046.tqpisa (mgr.14217) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi100 on smithi100: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-922469d0-8d32-11ee-95a2-87774f69a715-ceph-
    exporter-smithi100" in cluster log

[7467737]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic overrides/ignorelist}
-----------------------------------------------------------------
time:   00:47:08
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467737/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467737/

    "/var/log/ceph/065b7222-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:38:13.285+0000 7fcabc03a700  0 log_channel(cluster)
    log [WRN] : Health check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7467738]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only overrides/ignorelist start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:11:54
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467738/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467738/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e1ecf42717224831bd36f5a6f5fcec42

    Command failed on smithi017 with status 1: 'sudo yum -y install ceph'

[7467740]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:15:47
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467740/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467740/

    "/var/log/ceph/366d6fba-8d32-11ee-95a2-87774f69a715/ceph-
    mon.smithi049.log:2023-11-27T14:37:57.889+0000 7f8a7a2d4700 10
    mon.smithi049@0(leader).log v77  logging 2023-11-27T14:37:57.778555+0000
    mgr.smithi049.rkdusc (mgr.14215) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi181 on smithi181: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-366d6fba-8d32-11ee-95a2-87774f69a715-ceph-
    exporter-smithi181" in cluster log

[7467741]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:40
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467741/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467741/

    "/var/log/ceph/83b4b0b0-8d34-11ee-95a2-87774f69a715/ceph-
    mon.smithi001.log:2023-11-27T14:57:49.691+0000 7fd46f25b700  0
    log_channel(cluster) log [WRN] : Replacing daemon
    mds.cephfs.smithi001.nshvny as rank 0 with standby daemon
    mds.cephfs.smithi162.wzoysn" in cluster log

[7467742]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all overrides/ignorelist}
-----------------------------------------------------------------
time:   00:17:36
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467742/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467742/

    "/var/log/ceph/1e1b9b04-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi112.log:2023-11-27T14:30:43.060+0000 7fbd6d1fc700 10
    mon.smithi112@0(leader).log v90  logging 2023-11-27T14:30:42.495769+0000
    mgr.smithi112.wbftnn (mgr.14215) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi175 on smithi175: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-1e1b9b04-8d31-11ee-95a2-87774f69a715-ceph-
    exporter-smithi175" in cluster log

[7467743]  orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:16:24
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467743/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467743/

    "/var/log/ceph/b6898cce-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:41:16.441+0000 7fcbac0a5700  0 log_channel(cluster)
    log [WRN] :     mon.c (rank 2) addr
    [v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0] is down (out of quorum)" in
    cluster log

[7467744]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:50
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467744/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467744/

    "/var/log/ceph/c49228e0-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi073.log:2023-11-27T14:35:46.208+0000 7f83332c6700 10
    mon.smithi073@0(leader).log v100  logging 2023-11-27T14:35:45.467096+0000
    mgr.smithi073.bnbhsk (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi161 on smithi161: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-c49228e0-8d31-11ee-95a2-87774f69a715-ceph-
    exporter-smithi161" in cluster log

[7467745]  orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only overrides/ignorelist root}
-----------------------------------------------------------------
time:   00:11:44
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467745/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01bc1409820142a784b753698a77dfc5

    Command failed on smithi193 with status 1: 'sudo yum -y install ceph'

[7467746]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async overrides/ignorelist start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:11:49
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467746/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467746/
sentry: https://sentry.ceph.com/organizations/ceph/?query=738b958ea3d84ff79985e437bb235e3e

    Command failed on smithi089 with status 1: 'sudo yum -y install ceph'

[7467747]  orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic overrides/ignorelist task/test_adoption}
-----------------------------------------------------------------
time:   00:10:53
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467747/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d480242c97b64e658d37d7e11a25f95c

    Command failed (workunit test cephadm/test_adoption.sh) on smithi003 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a501ccc0f4e7124633d52671cd089579114504d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'

[7467748]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:17:17
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467748/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467748/

    "/var/log/ceph/d632e9cc-8d31-11ee-95a2-87774f69a715/ceph-
    mon.smithi105.log:2023-11-27T14:35:48.626+0000 7ff2e4244700 10
    mon.smithi105@0(leader).log v88  logging 2023-11-27T14:35:47.879954+0000
    mgr.smithi105.lccril (mgr.14213) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi144 on smithi144: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-d632e9cc-8d31-11ee-95a2-87774f69a715-ceph-
    exporter-smithi144" in cluster log

[7467751]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:58
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467751/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467751/

    "/var/log/ceph/24369c18-8d32-11ee-95a2-87774f69a715/ceph-
    mon.smithi005.log:2023-11-27T14:39:09.275+0000 7fef0f483700 10
    mon.smithi005@0(leader).log v107  logging 2023-11-27T14:39:09.089167+0000
    mgr.smithi005.ckxavt (mgr.14195) 17 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi052 on smithi052: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-24369c18-8d32-11ee-95a2-87774f69a715-ceph-
    exporter-smithi052" in cluster log

[7467752]  orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli overrides/ignorelist}
-----------------------------------------------------------------
time:   00:11:46
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467752/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a327ea40590248c3ac14423f23a49c86

    Command failed on smithi164 with status 1: 'sudo yum -y install ceph'

[7467754]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:14:35
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467754/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467754/

    "/var/log/ceph/e5bce716-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:43:32.890+0000 7f6041eff700  0 log_channel(cluster)
    log [WRN] : Health check failed: 1 failed cephadm daemon(s)
    (CEPHADM_FAILED_DAEMON)" in cluster log

[7467755]  orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only overrides/ignorelist root}
-----------------------------------------------------------------
time:   00:11:29
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467755/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467755/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44450e37f827469ba8a6cb26d61032c2

    Command failed on smithi027 with status 1: 'sudo yum -y install ceph'

[7467757]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:15:03
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467757/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467757/

    "/var/log/ceph/166a167c-8d33-11ee-95a2-87774f69a715/ceph-
    mon.smithi078.log:2023-11-27T14:44:19.094+0000 7fc18f56c700 10
    mon.smithi078@0(leader).log v78  logging 2023-11-27T14:44:18.750547+0000
    mgr.smithi078.rsixte (mgr.14215) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi148 on smithi148: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-166a167c-8d33-11ee-95a2-87774f69a715-ceph-
    exporter-smithi148" in cluster log

[7467758]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity overrides/ignorelist}
-----------------------------------------------------------------
time:   00:38:22
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467758/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467758/

    "/var/log/ceph/e9f64a20-8d32-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:43:19.741+0000 7f5b0a481700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum)"
    in cluster log

[7467759]  orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:49
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467759/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467759/

    "/var/log/ceph/f3cea222-8d32-11ee-95a2-87774f69a715/ceph-
    mon.smithi040.log:2023-11-27T14:45:19.547+0000 7f8cd3468700 10
    mon.smithi040@0(leader).log v105  logging 2023-11-27T14:45:18.800316+0000
    mgr.smithi040.vxevao (mgr.14197) 17 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi119 on smithi119: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-f3cea222-8d32-11ee-95a2-87774f69a715-ceph-
    exporter-smithi119" in cluster log

[7467761]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only overrides/ignorelist start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:35:55
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467761/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467761/

    "/var/log/ceph/0392cb28-8d35-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:09:00.497+0000 7f7e6f58d700 10 mon.a@0(leader).log
    v533  logging 2023-11-27T15:08:59.879469+0000 mgr.x (mgr.24466) 28 :
    cephadm [ERR] Non-zero return from ['ceph', '-k',
    '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x',
    'rotate-key', '-i', '-']: 2023-11-27T15:08:59.872+0000 7efd837fe700  1 --
    172.21.15.188:0/4189649617 <== mon.1 v2:172.21.15.188:3300/0 4 ====
    auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0)
    0x7efd74003770 con 0x7efd84108ca0" in cluster log

[7467762]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:12:48
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467762/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467762/

    "/var/log/ceph/ef5b28c6-8d39-11ee-95a2-87774f69a715/ceph-
    mon.smithi071.log:2023-11-27T15:39:47.371+0000 7f9aa80ea700  0
    log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN 1 filesystem
    with deprecated feature inline_data" in cluster log

[7467763]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:26:37
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467763/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467763/

    "/var/log/ceph/23ff7ce4-8d35-11ee-95a2-87774f69a715/ceph-
    mon.smithi131.log:2023-11-27T14:59:02.347+0000 7f0641d24700 10
    mon.smithi131@0(leader).log v73  logging 2023-11-27T14:59:02.041668+0000
    mgr.smithi131.cddsbi (mgr.14211) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi179 on smithi179: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-23ff7ce4-8d35-11ee-95a2-87774f69a715-ceph-
    exporter-smithi179" in cluster log

[7467765]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:20:50
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467765/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467765/

    "/var/log/ceph/836a8d56-8d33-11ee-95a2-87774f69a715/ceph-
    mon.smithi057.log:2023-11-27T14:48:25.389+0000 7fc311a1b700 10
    mon.smithi057@0(leader).log v98  logging 2023-11-27T14:48:25.246500+0000
    mgr.smithi057.uxlpbo (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi107 on smithi107: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-836a8d56-8d33-11ee-95a2-87774f69a715-ceph-
    exporter-smithi107" in cluster log

[7467766]  orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async overrides/ignorelist root}
-----------------------------------------------------------------
time:   00:46:37
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467766/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467766/

    "/var/log/ceph/04d6ec9a-8d34-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T14:52:20.805+0000 7f1159e19700  0 log_channel(cluster)
    log [WRN] :     mon.c (rank 2) addr
    [v2:172.21.15.90:3301/0,v1:172.21.15.90:6790/0] is down (out of quorum)" in
    cluster log

[7467767]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async overrides/ignorelist start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:22:58
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467767/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467767/

    "/var/log/ceph/97cd1f32-8d3a-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:49:59.999+0000 7fbf16710700  0 log_channel(cluster)
    log [WRN] :     application not enabled on pool 'ceph_test_rados_api_asio'"
    in cluster log

[7467768]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic overrides/ignorelist task/test_extra_daemon_features}
-----------------------------------------------------------------
time:   00:11:32
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467768/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467768/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0472d31b30b84986bb1eb870f94f712c

    Command failed on smithi142 with status 1: 'sudo yum -y install ceph'

[7467769]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:18:25
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467769/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467769/

    "/var/log/ceph/99eb888c-8d33-11ee-95a2-87774f69a715/ceph-
    mon.smithi088.log:2023-11-27T14:48:25.406+0000 7fde807db700 10
    mon.smithi088@0(leader).log v82  logging 2023-11-27T14:48:24.497661+0000
    mgr.smithi088.pxlimz (mgr.14213) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi092 on smithi092: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-99eb888c-8d33-11ee-95a2-87774f69a715-ceph-
    exporter-smithi092" in cluster log

[7467770]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag overrides/ignorelist}
-----------------------------------------------------------------
time:   00:34:24
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467770/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467770/

    "/var/log/ceph/abe03a26-8d36-11ee-95a2-87774f69a715/ceph-
    mon.smithi017.log:2023-11-27T15:10:08.573+0000 7f2177e34700 10
    mon.smithi017@0(leader).log v86  logging 2023-11-27T15:10:07.612414+0000
    mgr.smithi017.csknab (mgr.14217) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi111 on smithi111: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-abe03a26-8d36-11ee-95a2-87774f69a715-ceph-
    exporter-smithi111" in cluster log

[7467771]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:20:50
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467771/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467771/

    "/var/log/ceph/ec98af74-8d33-11ee-95a2-87774f69a715/ceph-
    mon.c.log:2023-11-27T14:53:37.688+0000 7fe6f19d0700  7 mon.c@2(peon).log
    v203 update_from_paxos applying incremental log 203
    2023-11-27T14:53:36.798111+0000 mon.a (mon.0) 726 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7467772]  orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity overrides/ignorelist task/test_host_drain}
-----------------------------------------------------------------
time:   00:23:50
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467772/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467772/

    "/var/log/ceph/1ca71b60-8d34-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:02:29.415+0000 7f42e48ab700 10 mon.a@0(leader).log
    v446  logging 2023-11-27T15:02:29.265649+0000 mgr.a (mgr.14152) 463 :
    cephadm [ERR] host smithi184 `cephadm ceph-volume` failed: Cannot decode
    JSON: " in cluster log

[7467773]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:21:45
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467773/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467773/

    "/var/log/ceph/ee96e052-8d33-11ee-95a2-87774f69a715/ceph-
    mon.smithi028.log:2023-11-27T14:52:00.846+0000 7f818fb6b700 10
    mon.smithi028@0(leader).log v100  logging 2023-11-27T14:51:59.899818+0000
    mgr.smithi028.zebsby (mgr.14193) 16 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi140 on smithi140: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-ee96e052-8d33-11ee-95a2-87774f69a715-ceph-
    exporter-smithi140" in cluster log

[7467774]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only overrides/ignorelist start tasks/rados_python}
-----------------------------------------------------------------
time:   00:11:01
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467774/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467774/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54af598e1aec4a78b55ac61975aa98b9

    Command failed on smithi150 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest
    python3-pytest'

[7467775]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:20:25
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467775/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467775/

    "/var/log/ceph/87f606b8-8d3b-11ee-95a2-87774f69a715/ceph-
    mon.smithi059.log:2023-11-27T15:47:54.920+0000 7f54129ef700  0
    log_channel(cluster) log [WRN] : Replacing daemon
    mds.cephfs.smithi059.ejobnm as rank 0 with standby daemon
    mds.cephfs.smithi167.gwscqw" in cluster log

[7467776]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:37:14
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467776/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467776/

    "/var/log/ceph/4804b418-8d37-11ee-95a2-87774f69a715/ceph-
    mon.smithi099.log:2023-11-27T15:14:18.552+0000 7f0f3fa89700 10
    mon.smithi099@0(leader).log v80  logging 2023-11-27T15:14:18.304648+0000
    mgr.smithi099.fjfzkz (mgr.14215) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi120 on smithi120: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-4804b418-8d37-11ee-95a2-87774f69a715-ceph-
    exporter-smithi120" in cluster log

[7467777]  orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:34:48
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467777/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467777/

    "/var/log/ceph/9cfabc42-8d37-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:17:05.480+0000 7fe9d5443700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] is down (out of quorum)"
    in cluster log

[7467778]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only overrides/ignorelist root}
-----------------------------------------------------------------
time:   01:27:30
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467778/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467778/

    "/var/log/ceph/f59b0a74-8d3b-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:53:42.838+0000 7f776191a700  0 log_channel(cluster)
    log [WRN] : Health check failed: Degraded data redundancy: 109/735 objects
    degraded (14.830%), 26 pgs degraded (PG_DEGRADED)" in cluster log

[7467779]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic overrides/ignorelist}
-----------------------------------------------------------------
time:   00:58:55
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467779/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467779/

    "/var/log/ceph/128c277e-8d39-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:35:09.582+0000 7f3b57889700  0 log_channel(cluster)
    log [WRN] :     mon.b (rank 2) addr
    [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum)"
    in cluster log

[7467781]  orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only overrides/ignorelist start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:11:32
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467781/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467781/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a7319d3a0b7c47428fbf2f6a09793c71

    Command failed on smithi152 with status 1: 'sudo yum -y install ceph'

[7467782]  orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:37:39
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467782/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467782/

    "/var/log/ceph/8d1f2970-8d37-11ee-95a2-87774f69a715/ceph-
    mon.smithi106.log:2023-11-27T15:16:21.495+0000 7f7cbfad9700 10
    mon.smithi106@0(leader).log v78  logging 2023-11-27T15:16:21.330854+0000
    mgr.smithi106.fuyeea (mgr.14215) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi195 on smithi195: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-8d1f2970-8d37-11ee-95a2-87774f69a715-ceph-
    exporter-smithi195" in cluster log

[7467783]  orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity overrides/ignorelist task/test_monitoring_stack_basic}
-----------------------------------------------------------------
time:   01:10:26
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467783/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467783/

    "/var/log/ceph/4db1f6aa-8d3c-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:50:25.583+0000 7f675d17e700  0 log_channel(cluster)
    log [WRN] : Health check failed: 1 failed cephadm daemon(s)
    (CEPHADM_FAILED_DAEMON)" in cluster log

[7467784]  orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait overrides/ignorelist}
-----------------------------------------------------------------
time:   00:36:09
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467784/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467784/

    "/var/log/ceph/02467564-8d38-11ee-95a2-87774f69a715/ceph-
    mon.smithi112.log:2023-11-27T15:20:08.116+0000 7fd9b395b700 10
    mon.smithi112@0(leader).log v89  logging 2023-11-27T15:20:07.912075+0000
    mgr.smithi112.ftxegi (mgr.14225) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi175 on smithi175: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-02467564-8d38-11ee-95a2-87774f69a715-ceph-
    exporter-smithi175" in cluster log

[7467785]  orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity overrides/ignorelist start}
-----------------------------------------------------------------
time:   00:19:22
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467785/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467785/

    "/var/log/ceph/be21eaf6-8d34-11ee-95a2-87774f69a715/ceph-
    mon.c.log:2023-11-27T14:56:14.188+0000 7ff8e7e4b700  7
    mon.c@2(synchronizing).log v63 update_from_paxos applying incremental log
    62 2023-11-27T14:56:12.206253+0000 mon.a (mon.0) 176 : cluster [WRN]
    mon.c (rank 2) addr [v2:172.21.15.61:3301/0,v1:172.21.15.61:6790/0] is down
    (out of quorum)" in cluster log

[7467786]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:33
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467786/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467786/

    "/var/log/ceph/b9aa77c2-8d34-11ee-95a2-87774f69a715/ceph-
    mon.smithi137.log:2023-11-27T14:56:42.859+0000 7f1996762700 10
    mon.smithi137@0(leader).log v97  logging 2023-11-27T14:56:42.746939+0000
    mgr.smithi137.bklslj (mgr.14217) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi172 on smithi172: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-b9aa77c2-8d34-11ee-95a2-87774f69a715-ceph-
    exporter-smithi172" in cluster log

[7467787]  orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only overrides/ignorelist root}
-----------------------------------------------------------------
time:   01:21:08
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467787/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467787/

    "/var/log/ceph/861b2be6-8d3d-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:58:25.315+0000 7faf333e6700  0 log_channel(cluster)
    log [WRN] :     mon.c (rank 2) addr
    [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] is down (out of quorum)"
    in cluster log

[7467788]  orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async overrides/ignorelist start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:44:19
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467788/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467788/

    "/var/log/ceph/1532d6d4-8d35-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T15:19:59.996+0000 7f7dd87a3700  0 log_channel(cluster)
    log [WRN] :     application not enabled on pool 'ceph_test_rados_api_asio'"
    in cluster log

[7467789]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic overrides/ignorelist task/test_orch_cli}
-----------------------------------------------------------------
time:   00:11:56
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467789/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6d44cc3b93f44b62a6d16660547bf1c5

    Command failed on smithi156 with status 1: 'sudo yum -y install ceph'

[7467790]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:09
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467790/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467790/

    "/var/log/ceph/f41b7c3a-8d34-11ee-95a2-87774f69a715/ceph-
    mon.smithi050.log:2023-11-27T14:58:12.096+0000 7f1c4aa22700 10
    mon.smithi050@0(leader).log v89  logging 2023-11-27T14:58:11.679687+0000
    mgr.smithi050.mvjpkr (mgr.14211) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi164 on smithi164: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-f41b7c3a-8d34-11ee-95a2-87774f69a715-ceph-
    exporter-smithi164" in cluster log

[7467791]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:15:28
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467791/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467791/

    "/var/log/ceph/bf259db8-8d3d-11ee-95a2-87774f69a715/ceph-
    mon.smithi098.log:2023-11-27T16:03:46.781+0000 7f39e986a700  0
    log_channel(cluster) log [WRN] : Replacing daemon
    mds.cephfs.smithi098.mhedjt as rank 0 with standby daemon
    mds.cephfs.smithi098.krrezx" in cluster log

[7467792]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final overrides/ignorelist}
-----------------------------------------------------------------
time:   00:19:49
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467792/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467792/

    "/var/log/ceph/0ee01882-8d35-11ee-95a2-87774f69a715/ceph-
    mon.smithi003.log:2023-11-27T15:00:19.060+0000 7f64d2aa8700 10
    mon.smithi003@0(leader).log v110  logging 2023-11-27T15:00:18.184848+0000
    mgr.smithi003.ahfggm (mgr.14195) 17 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi129 on smithi129: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 1 from /usr/bin/docker container inspect
    --format {{.State.Status}} ceph-0ee01882-8d35-11ee-95a2-87774f69a715-ceph-
    exporter-smithi129" in cluster log

[7467793]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity overrides/ignorelist task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:13:04
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467793/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8ebf7c621fa4b1088f20921287e89f3

    Command failed on smithi173 with status 1: 'sudo yum -y install ceph'

[7467794]  orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only overrides/ignorelist start tasks/rados_python}
-----------------------------------------------------------------
time:   01:08:39
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467794/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467794/

    "/var/log/ceph/591e4c62-8d3e-11ee-95a2-87774f69a715/ceph-
    mon.a.log:2023-11-27T16:09:59.999+0000 7fa8cca43700  0 log_channel(cluster)
    log [WRN] :     application not enabled on pool 'rbd'" in cluster log

[7467795]  orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate overrides/ignorelist}
-----------------------------------------------------------------
time:   00:21:57
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467795/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467795/

    "/var/log/ceph/861fc1fe-8d35-11ee-95a2-87774f69a715/ceph-
    mon.smithi055.log:2023-11-27T15:02:52.951+0000 7fd207a9a700 10
    mon.smithi055@0(leader).log v104  logging 2023-11-27T15:02:52.467956+0000
    mgr.smithi055.hoocts (mgr.14219) 15 : cephadm [ERR] Failed while placing
    ceph-exporter.smithi144 on smithi144: cephadm exited with an error code: 1,
    stderr: Non-zero exit code 125 from /usr/bin/podman container inspect
    --format {{.State.Status}} ceph-861fc1fe-8d35-11ee-95a2-87774f69a715-ceph-
    exporter-smithi144" in cluster log



Dead
=================================================================
[7467704]  orch:cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467704/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467704/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds

[7467739]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity overrides/ignorelist task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467739/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467739/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Pass
=================================================================
[7467701] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity overrides/ignorelist start}
time:   00:20:33
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467701/

[7467705] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/off orchestrator_cli overrides/ignorelist}
time:   00:14:07
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467705/

[7467706] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:56:53
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467706/

[7467707] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final overrides/ignorelist}
time:   00:22:08
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467707/

[7467714] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic overrides/ignorelist task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   00:24:37
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467714/

[7467720] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic overrides/ignorelist start}
time:   00:18:52
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467720/

[7467732] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity overrides/ignorelist start}
time:   00:26:37
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467732/

[7467749] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity overrides/ignorelist task/test_ca_signed_key}
time:   00:17:12
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467749/

[7467750] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only overrides/ignorelist start tasks/rados_python}
time:   00:32:09
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467750/

[7467753] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final overrides/ignorelist}
time:   00:19:12
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467753/

[7467756] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic overrides/ignorelist task/test_cephadm}
time:   00:18:30
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467756/

[7467760] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity overrides/ignorelist start}
time:   00:16:16
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467760/

[7467764] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity overrides/ignorelist task/test_cephadm_repos}
time:   00:12:43
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467764/

[7467780] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic overrides/ignorelist task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}}
time:   01:05:51
info:   https://pulpito.ceph.com/adking-2023-11-27_13:10:55-orch:cephadm-wip-adk3-testing-2023-11-26-1627-distro-default-smithi/7467780/