2023-11-16T17:22:08.142 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T17:22:08.142 INFO:teuthology.results:subset: '1/14' 2023-11-16T17:22:08.142 INFO:teuthology.results:seed: '3445' 2023-11-16T17:22:21.294 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T17:22:21.294 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-16T18:13:24.946 INFO:teuthology.results:Tests finished! gathering results... 2023-11-16T18:13:27.862 INFO:scrape:Found 19 jobs 2023-11-16T18:13:29.779 INFO:scrape:Found 4 distinct failure reasons 2023-11-16T18:13:29.779 INFO:scrape:Failure: Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fc99696c-84a6-11ee-8dbe-212e2dc638e7 -e sha1=50bc6740f385afe4f36770e739358479a73d8add -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-11-16T18:13:29.780 INFO:scrape:3 jobs: ['7460926', '7460934', '7460935'] 2023-11-16T18:13:29.780 INFO:scrape:suites intersection: [] 2023-11-16T18:13:29.780 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '3-upgrade-with-workload', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic', 'mon_election/classic}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}'] 2023-11-16T18:13:29.780 INFO:scrape: 2023-11-16T18:13:29.780 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi137 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bc6740f385afe4f36770e739358479a73d8add TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' 2023-11-16T18:13:29.780 INFO:scrape:2 jobs: ['7460942', '7460929'] 2023-11-16T18:13:29.780 INFO:scrape:suites intersection: ['task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-16T18:13:29.780 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-11-16T18:13:29.781 INFO:scrape: 2023-11-16T18:13:29.781 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi176.front.sepia.ceph.com: ['type=AVC msg=audit(1700156813.508:21449): avc: denied { ioctl } for pid=121269 comm="iptables" path="/var/lib/containers/storage/overlay/13e391e31eadac1f2ffb214bcaaa21d291778060ea59f2d8745aa0a141be94eb/merged" dev="overlay" ino=2754482 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 2023-11-16T18:13:29.781 INFO:scrape:1 jobs: ['7460939'] 2023-11-16T18:13:29.781 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}'] 2023-11-16T18:13:29.781 INFO:scrape: 2023-11-16T18:13:29.781 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi091 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bc6740f385afe4f36770e739358479a73d8add TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-11-16T18:13:29.781 INFO:scrape:1 jobs: ['7460941'] 2023-11-16T18:13:29.781 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm_repos}'] 2023-11-16T18:13:29.781 INFO:scrape: 2023-11-16T18:13:29.784 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [7460926] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:32:38 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460926/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=083b650f6bf34bc89c2204c71e66cdfb Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fc99696c-84a6-11ee-8dbe-212e2dc638e7 -e sha1=50bc6740f385afe4f36770e739358479a73d8add -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7460929] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:19:03 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460929/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4c1ba0800524f5fa8ec7baf1183ff51 Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi002 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bc6740f385afe4f36770e739358479a73d8add TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' [7460934] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:19:14 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460934/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460934/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f7d5a34570e4edf948683a89eca90c2 Command failed on smithi063 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:50bc6740f385afe4f36770e739358479a73d8add shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49b4fd4c-84a7-11ee-8dbe-212e2dc638e7 -- ceph rgw realm bootstrap -i -' [7460935] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:41:07 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460935/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b28c4c53c4484f1580abcd91f5d4fb98 Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 435903e4-84a7-11ee-8dbe-212e2dc638e7 -e sha1=50bc6740f385afe4f36770e739358479a73d8add -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' [7460939] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:21:00 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460939/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460939/ SELinux denials found on ubuntu@smithi176.front.sepia.ceph.com: ['type=AVC msg=audit(1700156813.508:21449): avc: denied { ioctl } for pid=121269 comm="iptables" path="/var/lib/containers/storage/overlay/13e391e31eadac1f2 ffb214bcaaa21d291778060ea59f2d8745aa0a141be94eb/merged" dev="overlay" ino=2754482 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] [7460941] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:41 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460941/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1eae3c11ef3c47db849aea67a9d01837 Command failed (workunit test cephadm/test_repos.sh) on smithi091 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bc6740f385afe4f36770e739358479a73d8add TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7460942] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:17:35 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460942/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f5a45f815b04bd7b9df46cb8df9b899 Command failed (workunit test cephadm/test_iscsi_setup.sh) on smithi137 with status 24: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bc6740f385afe4f36770e739358479a73d8add TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' Pass ================================================================= [7460924] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:20:36 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460924/ [7460925] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:11 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460925/ [7460927] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:48 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460927/ [7460928] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:23:18 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460928/ [7460930] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:15:38 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460930/ [7460931] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:20:07 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460931/ [7460932] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:23:53 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460932/ [7460933] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:36:33 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460933/ [7460936] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:20:09 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460936/ [7460937] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:17:13 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460937/ [7460938] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:32:09 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460938/ [7460940] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:38:55 info: https://pulpito.ceph.com/adking-2023-11-16_17:22:00-orch:cephadm-wip-adk-testing-2023-11-16-0243-distro-default-smithi/7460940/