2023-01-23T04:36:50.212 INFO:root:teuthology version: 0.0.1.dev20+g5abecab 2023-01-23T04:36:50.213 INFO:teuthology.results:subset: '1/10' 2023-01-23T04:36:50.213 INFO:teuthology.results:seed: '3761' 2023-01-23T04:37:05.370 INFO:root:teuthology version: 0.0.1.dev20+g5abecab 2023-01-23T04:37:05.371 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-23T05:44:09.792 INFO:teuthology.results:Tests finished! gathering results... 2023-01-23T05:44:10.148 INFO:scrape:Found 15 jobs 2023-01-23T05:44:10.396 INFO:scrape:Found 3 distinct failure reasons 2023-01-23T05:44:10.397 INFO:scrape:Failure: Command failed on smithi093 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:749c36e80e1d077a46b710d0776d199145af6283 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 28e287a4-9ada-11ed-9e55-001a4aab830c -- bash -c \'ceph mon dump --format json | jq -e \'"\'"\'.mons | .[] | select(.name == "host.a") | .crush_location == "{datacenter=a}"\'"\'"\'\'' 2023-01-23T05:44:10.397 INFO:scrape:4 jobs: ['7133581', '7133590', '7133576', '7133589'] 2023-01-23T05:44:10.397 INFO:scrape:suites intersection: ['agent/on'] 2023-01-23T05:44:10.398 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_set_mon_crush_locations}'] 2023-01-23T05:44:10.398 INFO:scrape: 2023-01-23T05:44:10.398 INFO:scrape:Failure: Command failed on smithi072 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-01-23T05:44:10.399 INFO:scrape:1 jobs: ['7133588'] 2023-01-23T05:44:10.399 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}'] 2023-01-23T05:44:10.400 INFO:scrape: 2023-01-23T05:44:10.400 INFO:scrape:Failure: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c0e3118-9ada-11ed-9e55-001a4aab830c -e sha1=749c36e80e1d077a46b710d0776d199145af6283 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' 2023-01-23T05:44:10.400 INFO:scrape:1 jobs: ['7133579'] 2023-01-23T05:44:10.401 INFO:scrape:suites: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools'] 2023-01-23T05:44:10.401 INFO:scrape: 2023-01-23T05:44:10.402 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7133576] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:19:23 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133576/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133576/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb8167a17d3942acac1a65dd9232f213 Command failed on smithi080 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:749c36e80e1d077a46b710d0776d199145af6283 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8938d180-9ada-11ed-9e55-001a4aab830c -- bash -c \'ceph mon dump --format json | jq -e \'"\'"\'.mons | .[] | select(.name == "host.a") | .crush_location == "{datacenter=a}"\'"\'"\'\'' [7133579] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:55:40 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133579/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133579/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da468828fe324e85aeb52cf3f430bee0 Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c0e3118-9ada-11ed-9e55-001a4aab830c -e sha1=749c36e80e1d077a46b710d0776d199145af6283 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' [7133581] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:20:37 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133581/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133581/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de382383a31a4a5ba2813d7a915b38fc Command failed on smithi093 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:749c36e80e1d077a46b710d0776d199145af6283 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 28e287a4-9ada-11ed-9e55-001a4aab830c -- bash -c \'ceph mon dump --format json | jq -e \'"\'"\'.mons | .[] | select(.name == "host.a") | .crush_location == "{datacenter=a}"\'"\'"\'\'' [7133588] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}} ----------------------------------------------------------------- time: 00:06:34 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133588/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133588/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4cd4b92c47643f2842fdc45c03f8dc5 Command failed on smithi072 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7133589] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:55:37 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133589/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133589/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8787dbbe330846a09cad324ae25aafa8 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d4d8658-9ada-11ed-9e55-001a4aab830c -e sha1=749c36e80e1d077a46b710d0776d199145af6283 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' [7133590] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:20:01 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133590/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133590/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9796874c3352480ab5142c032df7af8c Command failed on smithi037 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:749c36e80e1d077a46b710d0776d199145af6283 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9ff14fb0-9ada-11ed-9e55-001a4aab830c -- bash -c \'ceph mon dump --format json | jq -e \'"\'"\'.mons | .[] | select(.name == "host.a") | .crush_location == "{datacenter=a}"\'"\'"\'\'' Pass ================================================================= [7133577] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:30:28 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133577/ [7133578] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:04 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133578/ [7133580] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:15:04 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133580/ [7133582] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:27:46 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133582/ [7133583] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:08 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133583/ [7133584] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:32:27 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133584/ [7133585] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm} time: 00:18:01 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133585/ [7133586] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:37:20 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133586/ [7133587] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos} time: 00:09:15 info: http://pulpito.ceph.com/adking-2023-01-23_04:36:45-orch:cephadm-wip-adk-testing-2023-01-22-1523-distro-default-smithi/7133587/