2023-11-14T13:06:22.457 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T13:06:22.459 INFO:teuthology.results:subset: '1/14' 2023-11-14T13:06:22.459 INFO:teuthology.results:seed: '2762' 2023-11-14T13:11:00.350 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T13:11:00.353 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-14T14:14:05.088 INFO:teuthology.results:Tests finished! gathering results... 2023-11-14T14:14:05.289 INFO:scrape:Found 7 jobs 2023-11-14T14:14:05.501 INFO:scrape:Found 4 distinct failure reasons 2023-11-14T14:14:05.501 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi129 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=041583cc2585363a9346df3fc9233614e8c64dad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' 2023-11-14T14:14:05.502 INFO:scrape:2 jobs: ['7457124', '7457120'] 2023-11-14T14:14:05.502 INFO:scrape:suites intersection: ['task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-14T14:14:05.503 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-14T14:14:05.503 INFO:scrape: 2023-11-14T14:14:05.503 INFO:scrape:Failure: Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b305b2fc-82f2-11ee-8dbe-212e2dc638e7 -e sha1=041583cc2585363a9346df3fc9233614e8c64dad -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-11-14T14:14:05.504 INFO:scrape:1 jobs: ['7457119'] 2023-11-14T14:14:05.504 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-11-14T14:14:05.505 INFO:scrape: 2023-11-14T14:14:05.505 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=041583cc2585363a9346df3fc9233614e8c64dad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-11-14T14:14:05.505 INFO:scrape:1 jobs: ['7457123'] 2023-11-14T14:14:05.506 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm_repos}'] 2023-11-14T14:14:05.506 INFO:scrape: 2023-11-14T14:14:05.507 INFO:scrape:Failure: Command failed on smithi002 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:041583cc2585363a9346df3fc9233614e8c64dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be9e7e00-82f2-11ee-8dbe-212e2dc638e7 -- ceph rgw realm bootstrap -i -' 2023-11-14T14:14:05.507 INFO:scrape:1 jobs: ['7457121'] 2023-11-14T14:14:05.507 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}'] 2023-11-14T14:14:05.508 INFO:scrape: 2023-11-14T14:14:05.508 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7457119] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:36:43 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457119/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457119/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af3a8785d95f4d4f89b57baa2055f817 Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b305b2fc-82f2-11ee-8dbe-212e2dc638e7 -e sha1=041583cc2585363a9346df3fc9233614e8c64dad -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7457120] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:23:04 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457120/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457120/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfec123783784125a07e010ee9ea3a96 Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi087 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=041583cc2585363a9346df3fc9233614e8c64dad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' [7457121] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:23:04 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457121/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4fe3e6443bc4d92998e0114099efcba Command failed on smithi002 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:041583cc2585363a9346df3fc9233614e8c64dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be9e7e00-82f2-11ee-8dbe-212e2dc638e7 -- ceph rgw realm bootstrap -i -' [7457123] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:12:18 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457123/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a024f28d9e8a42ddb3f1e5faf0bc9e9c Command failed (workunit test cephadm/test_repos.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=041583cc2585363a9346df3fc9233614e8c64dad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7457124] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:22:31 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457124/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f6da17b1f2844d89f8ee2a99db2a6fd Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi129 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=041583cc2585363a9346df3fc9233614e8c64dad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' Pass ================================================================= [7457122] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:52:24 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457122/ [7457125] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} time: 00:22:12 info: https://pulpito.ceph.com/adking-2023-11-14_13:06:17-orch:cephadm-wip-adk-testing-2023-11-13-1853-distro-default-smithi/7457125/