2023-11-08T07:46:47.031 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T07:46:47.033 INFO:teuthology.results:subset: '1/14' 2023-11-08T07:46:47.034 INFO:teuthology.results:seed: '4403' 2023-11-08T10:02:42.975 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T10:02:42.994 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-08T10:49:46.964 INFO:teuthology.results:Tests finished! gathering results... 2023-11-08T10:49:52.989 INFO:scrape:Found 98 jobs 2023-11-08T10:49:54.683 INFO:scrape:Found 6 distinct failure reasons 2023-11-08T10:49:54.684 INFO:scrape:Failure: Command failed on smithi187 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c38459e2-7e0d-11ee-8dbb-212e2dc638e7 -e sha1=2197ffb6d18705999f38c4c20fe51351c31e5ad7 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-11-08T10:49:54.684 INFO:scrape:1 jobs: ['7451636'] 2023-11-08T10:49:54.685 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-11-08T10:49:54.685 INFO:scrape: 2023-11-08T10:49:54.686 INFO:scrape:Failure: Command failed on smithi038 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2197ffb6d18705999f38c4c20fe51351c31e5ad7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71640d68-7e0f-11ee-8dbb-212e2dc638e7 -- bash -c \'set -ex\nHOSTNAMES=$(ceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\')\nfor host in $HOSTNAMES; do\n # find the hostname for "host.c" which will have no mgr\n HAS_MGRS=$(ceph orch ps --hostname ${host} --format json | jq \'"\'"\'any(.daemon_type == "mgr")\'"\'"\')\n if [ "$HAS_MGRS" == "false" ]; then\n HOST_C="${host}"\n fi\ndone\n# now drain that host\nceph orch host drain $HOST_C --zap-osd-devices\n# wait for drain to complete\nHOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\nwhile [ "$HOST_C_DAEMONS" != "No daemons reported" ]; do\n sleep 15\n HOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\ndone\n# we want to check the ability to remove the host from\n# the CRUSH map, so we should first verify the host is in\n# the CRUSH map.\nceph osd getcrushmap -o compiled-crushmap\ncrushtool -d compiled-crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif ! grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Expected to see $HOST_C in CRUSH map. Saw:\\n\\n$CRUSH_MAP"\n exit 1\nfi\n# One last thing to worry about before removing the host\n# is that the teuthology test tends to put the explicit\n# hostnames in the placement for the mon service.\n# We want to make sure we can remove without providing\n# --force and there is a check for the host being removed\n# being listed explicitly in the placements. Therefore,\n# we should remove it from the mon placement.\nceph orch ls mon --export > mon.yaml\nsed /"$HOST_C"/d mon.yaml > mon_adjusted.yaml\nceph orch apply -i mon_adjusted.yaml\n# If the drain was successful, we should be able to remove the\n# host without force with no issues. If there are still daemons\n# we will get a response telling us to drain the host and a\n# non-zero return code\nceph orch host rm $HOST_C --rm-crush-entry\n# verify we\'"\'"\'ve successfully removed the host from the CRUSH map\nsleep 30\nceph osd getcrushmap -o compiled-crushmap\ncrushtool -d compiled-crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Saw $HOST_C in CRUSH map after it should have been removed.\\n\\n$CRUSH_MAP"\n exit 1\nfi\n\'' 2023-11-08T10:49:54.686 INFO:scrape:2 jobs: ['7451644', '7451709'] 2023-11-08T10:49:54.686 INFO:scrape:suites intersection: ['task/test_host_drain}'] 2023-11-08T10:49:54.687 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_host_drain}'] 2023-11-08T10:49:54.687 INFO:scrape: 2023-11-08T10:49:54.688 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-11-08T10:49:54.688 INFO:scrape:1 jobs: ['7451696'] 2023-11-08T10:49:54.688 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm_repos}'] 2023-11-08T10:49:54.689 INFO:scrape: 2023-11-08T10:49:54.689 INFO:scrape:Failure: Command failed on smithi003 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2197ffb6d18705999f38c4c20fe51351c31e5ad7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1463170c-7e1a-11ee-8dbb-212e2dc638e7 -- ceph rgw realm bootstrap -i -' 2023-11-08T10:49:54.690 INFO:scrape:1 jobs: ['7451669'] 2023-11-08T10:49:54.690 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}'] 2023-11-08T10:49:54.691 INFO:scrape: 2023-11-08T10:49:54.691 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' 2023-11-08T10:49:54.691 INFO:scrape:2 jobs: ['7451648', '7451715'] 2023-11-08T10:49:54.692 INFO:scrape:suites intersection: ['task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-08T10:49:54.692 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-08T10:49:54.693 INFO:scrape: 2023-11-08T10:49:54.693 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2023-11-08T10:49:54.694 INFO:scrape:1 jobs: ['7451699'] 2023-11-08T10:49:54.694 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_timeout}'] 2023-11-08T10:49:54.694 INFO:scrape: 2023-11-08T10:49:54.698 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 90 Fail ================================================================= [7451636] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:30:57 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451636/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451636/ sentry: https://sentry.ceph.com/organizations/ceph/?query=621515044dce4311ae8176aca5c04016 Command failed on smithi187 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c38459e2-7e0d-11ee-8dbb-212e2dc638e7 -e sha1=2197ffb6d18705999f38c4c20fe51351c31e5ad7 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7451644] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:21:46 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451644/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451644/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6457b4554d524662b73f37727b61d05e Command failed on smithi038 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:2197ffb6d18705999f38c4c20fe51351c31e5ad7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71640d68-7e0f-11ee-8dbb-212e2dc638e7 -- bash -c \'set -ex\nHOSTNAMES=$(ceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\')\nfor host in $HOSTNAMES; do\n # find the hostname for "host.c" which will have no mgr\n HAS_MGRS=$(ceph orch ps --hostname ${host} --format json | jq \'"\'"\'any(.daemon_type == "mgr")\'"\'"\')\n if [ "$HAS_MGRS" == "false" ]; then\n HOST_C="${host}"\n fi\ndone\n# now drain that host\nceph orch host drain $HOST_C --zap-osd-devices\n# wait for drain to complete\nHOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\nwhile [ "$HOST_C_DAEMONS" != "No daemons reported" ]; do\n sleep 15\n HOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\ndone\n# we want to check the ability to remove the host from\n# the CRUSH map, so we should first verify the host is in\n# the CRUSH map.\nceph osd getcrushmap -o compiled- crushmap\ncrushtool -d compiled-crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif ! grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Expected to see $HOST_C in CRUSH map. Saw:\\n\\n$CRUSH_MAP"\n exit 1\nfi\n# One last thing to worry about before removing the host\n# is that the teuthology test tends to put the explicit\n# hostnames in the placement for the mon service.\n# We want to make sure we can remove without providing\n# --force and there is a check for the host being removed\n# being listed explicitly in the placements. Therefore,\n# we should remove it from the mon placement.\nceph orch ls mon --export > mon.yaml\nsed /"$HOST_C"/d mon.yaml > mon_adjusted.yaml\nceph orch apply -i mon_adjusted.yaml\n# If the drain was successful, we should be able to remove the\n# host without force with no issues. If there are still daemons\n# we will get a response telling us to drain the host and a\n# non-zero return code\nceph orch host rm $HOST_C --rm-crush-entry\n# verify we\'"\'"\'ve successfully removed the host from the CRUSH map\nsleep 30\nceph osd getcrushmap -o compiled-crushmap\ncrushtool -d compiled- crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Saw $HOST_C in CRUSH map after it should have been removed.\\n\\n$CRUSH_MAP"\n exit 1\nfi\n\'' [7451648] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:18:10 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451648/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451648/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a51d7de2ed494a72bc8a4c6e3aed636c Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' [7451669] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:19:10 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451669/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b279b11890f4c4181c4d287666b77a4 Command failed on smithi003 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:2197ffb6d18705999f38c4c20fe51351c31e5ad7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1463170c-7e1a-11ee-8dbb-212e2dc638e7 -- ceph rgw realm bootstrap -i -' [7451696] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:20 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451696/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1997857c700d461bb228ff6431adc33b Command failed (workunit test cephadm/test_repos.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7451699] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451699/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e8140d773a84bbcbcab3df4275ae336 Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7451709] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:25:23 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451709/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451709/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3d38998577d4563bead7b4aa3bfd6a0 Command failed on smithi042 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:2197ffb6d18705999f38c4c20fe51351c31e5ad7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b6f578bc-7e1e-11ee-8dbb-212e2dc638e7 -- bash -c \'set -ex\nHOSTNAMES=$(ceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\')\nfor host in $HOSTNAMES; do\n # find the hostname for "host.c" which will have no mgr\n HAS_MGRS=$(ceph orch ps --hostname ${host} --format json | jq \'"\'"\'any(.daemon_type == "mgr")\'"\'"\')\n if [ "$HAS_MGRS" == "false" ]; then\n HOST_C="${host}"\n fi\ndone\n# now drain that host\nceph orch host drain $HOST_C --zap-osd-devices\n# wait for drain to complete\nHOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\nwhile [ "$HOST_C_DAEMONS" != "No daemons reported" ]; do\n sleep 15\n HOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\ndone\n# we want to check the ability to remove the host from\n# the CRUSH map, so we should first verify the host is in\n# the CRUSH map.\nceph osd getcrushmap -o compiled- crushmap\ncrushtool -d compiled-crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif ! grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Expected to see $HOST_C in CRUSH map. Saw:\\n\\n$CRUSH_MAP"\n exit 1\nfi\n# One last thing to worry about before removing the host\n# is that the teuthology test tends to put the explicit\n# hostnames in the placement for the mon service.\n# We want to make sure we can remove without providing\n# --force and there is a check for the host being removed\n# being listed explicitly in the placements. Therefore,\n# we should remove it from the mon placement.\nceph orch ls mon --export > mon.yaml\nsed /"$HOST_C"/d mon.yaml > mon_adjusted.yaml\nceph orch apply -i mon_adjusted.yaml\n# If the drain was successful, we should be able to remove the\n# host without force with no issues. If there are still daemons\n# we will get a response telling us to drain the host and a\n# non-zero return code\nceph orch host rm $HOST_C --rm-crush-entry\n# verify we\'"\'"\'ve successfully removed the host from the CRUSH map\nsleep 30\nceph osd getcrushmap -o compiled-crushmap\ncrushtool -d compiled- crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif grep -q "$HOST_C" <<< "$CRUSH_MAP"; then\n printf "Saw $HOST_C in CRUSH map after it should have been removed.\\n\\n$CRUSH_MAP"\n exit 1\nfi\n\'' [7451715] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:18:47 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451715/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61453a1717fa4ad78d843994d27a7c07 Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2197ffb6d18705999f38c4c20fe51351c31e5ad7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' Pass ================================================================= [7451632] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:22:07 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451632/ [7451633] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:22:02 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451633/ [7451634] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:22 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451634/ [7451635] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:20:34 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451635/ [7451637] orch:cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:37:27 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451637/ [7451638] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/off orchestrator_cli} time: 00:14:55 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451638/ [7451639] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:35:34 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451639/ [7451640] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:17:47 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451640/ [7451641] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:08 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451641/ [7451642] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:20 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451642/ [7451643] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:28:45 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451643/ [7451645] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:27 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451645/ [7451646] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:23:32 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451646/ [7451647] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 00:59:48 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451647/ [7451649] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:38:04 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451649/ [7451650] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:16:35 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451650/ [7451651] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:40:48 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451651/ [7451652] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:16 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451652/ [7451653] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:28 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451653/ [7451654] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:22:54 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451654/ [7451655] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:15:10 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451655/ [7451656] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:34:51 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451656/ [7451657] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:39:40 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451657/ [7451658] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:21:25 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451658/ [7451659] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli} time: 00:23:29 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451659/ [7451660] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:01 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451660/ [7451661] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:01 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451661/ [7451662] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:32:56 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451662/ [7451663] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:26:36 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451663/ [7451664] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:09 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451664/ [7451665] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:18:49 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451665/ [7451666] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:31 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451666/ [7451667] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:19:55 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451667/ [7451668] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:37:47 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451668/ [7451670] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:18:54 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451670/ [7451671] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:47:34 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451671/ [7451672] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:32:09 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451672/ [7451673] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:21:00 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451673/ [7451674] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:17:54 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451674/ [7451675] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:44 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451675/ [7451676] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:48 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451676/ [7451677] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:59 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451677/ [7451678] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:42 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451678/ [7451679] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:46:53 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451679/ [7451680] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:42:35 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451680/ [7451681] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:16:17 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451681/ [7451682] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:18:04 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451682/ [7451683] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:19:45 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451683/ [7451684] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:31:50 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451684/ [7451685] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:14 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451685/ [7451686] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli} time: 00:19:34 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451686/ [7451687] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:18:06 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451687/ [7451688] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:13:06 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451688/ [7451689] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} time: 00:20:29 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451689/ [7451690] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:57:17 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451690/ [7451691] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:14:51 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451691/ [7451692] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:38:15 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451692/ [7451693] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:05 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451693/ [7451694] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:12 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451694/ [7451695] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:26:27 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451695/ [7451697] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:45 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451697/ [7451698] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:41 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451698/ [7451700] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:21:04 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451700/ [7451701] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:45:47 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451701/ [7451702] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:29 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451702/ [7451703] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:19:22 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451703/ [7451704] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:18:19 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451704/ [7451705] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:51 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451705/ [7451706] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:21:13 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451706/ [7451707] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:22:16 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451707/ [7451708] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:55 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451708/ [7451710] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:01 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451710/ [7451711] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:49 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451711/ [7451712] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:38 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451712/ [7451713] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:38:07 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451713/ [7451714] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:30:12 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451714/ [7451716] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:31:16 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451716/ [7451717] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:09 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451717/ [7451718] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:35 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451718/ [7451719] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:19:42 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451719/ [7451720] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:24:17 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451720/ [7451721] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:20:00 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451721/ [7451722] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:35:56 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451722/ [7451723] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:43:56 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451723/ [7451724] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:19:21 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451724/ [7451725] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} time: 00:17:37 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451725/ [7451726] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:07 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451726/ [7451727] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:11 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451727/ [7451728] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon} time: 00:24:45 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451728/ [7451729] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:23:50 info: https://pulpito.ceph.com/adking-2023-11-08_01:34:19-orch:cephadm-wip-adk-testing-2023-11-07-1631-distro-default-smithi/7451729/