2023-11-20T19:36:57.151 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-20T19:36:57.151 INFO:teuthology.results:subset: '1/14' 2023-11-20T19:36:57.151 INFO:teuthology.results:seed: '5141' 2023-11-20T20:02:25.343 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-20T20:02:25.348 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-21T03:08:23.538 INFO:teuthology.results:Tests finished! gathering results... 2023-11-21T03:08:23.867 INFO:scrape:Found 10 jobs 2023-11-21T03:08:24.784 INFO:scrape:Found 4 distinct failure reasons 2023-11-21T03:08:24.785 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi078 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3e320d455af86abe397f026166a45eba1e1ba6f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' 2023-11-21T03:08:24.785 INFO:scrape:2 jobs: ['7463668', '7463665'] 2023-11-21T03:08:24.785 INFO:scrape:suites intersection: ['task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-21T03:08:24.785 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-21T03:08:24.785 INFO:scrape: 2023-11-21T03:08:24.786 INFO:scrape:Failure: Command failed on smithi148 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef428c36-8815-11ee-959e-87774f69a715 -e sha1=3e320d455af86abe397f026166a45eba1e1ba6f8 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-11-21T03:08:24.786 INFO:scrape:1 jobs: ['7463661'] 2023-11-21T03:08:24.786 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-11-21T03:08:24.786 INFO:scrape: 2023-11-21T03:08:24.786 INFO:scrape:Failure: Command failed on smithi005 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3e320d455af86abe397f026166a45eba1e1ba6f8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 95d20e4a-87e0-11ee-959e-87774f69a715 -- ceph rgw realm bootstrap -i -' 2023-11-21T03:08:24.786 INFO:scrape:1 jobs: ['7463666'] 2023-11-21T03:08:24.786 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}'] 2023-11-21T03:08:24.787 INFO:scrape: 2023-11-21T03:08:24.787 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3e320d455af86abe397f026166a45eba1e1ba6f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-11-21T03:08:24.787 INFO:scrape:1 jobs: ['7463667'] 2023-11-21T03:08:24.787 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm_repos}'] 2023-11-21T03:08:24.787 INFO:scrape: 2023-11-21T03:08:24.788 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7463661] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 06:49:57 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463661/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07ce411c6ce74673967216956bf1f284 Command failed on smithi148 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ef428c36-8815-11ee-959e-87774f69a715 -e sha1=3e320d455af86abe397f026166a45eba1e1ba6f8 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7463665] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 06:28:51 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463665/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c67165504cb04305bd4fecb5dc31e65e Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi135 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3e320d455af86abe397f026166a45eba1e1ba6f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' [7463666] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 06:29:44 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463666/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463666/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98219b7bdac74862959ebd07b1f8b8c8 Command failed on smithi005 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:3e320d455af86abe397f026166a45eba1e1ba6f8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 95d20e4a-87e0-11ee-959e-87774f69a715 -- ceph rgw realm bootstrap -i -' [7463667] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 06:17:51 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463667/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463667/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2311a3c24e9f4f45b3377b8513143059 Command failed (workunit test cephadm/test_repos.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3e320d455af86abe397f026166a45eba1e1ba6f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7463668] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 06:25:19 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463668/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b74c6b5012e045f9b2e3388c474588ee Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi078 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3e320d455af86abe397f026166a45eba1e1ba6f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' Pass ================================================================= [7463659] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:21:18 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463659/ [7463660] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 06:34:54 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463660/ [7463662] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 07:00:56 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463662/ [7463663] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final} time: 06:32:23 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463663/ [7463664] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 06:25:28 info: https://pulpito.ceph.com/adking-2023-11-20_19:26:31-orch:cephadm-wip-adk-testing-2023-11-19-1956-distro-default-smithi/7463664/