2023-02-25T08:45:41.300 INFO:root:teuthology version: 0.0.1.dev43+ge06b2e0 2023-02-25T08:45:41.301 INFO:teuthology.results:subset: '1/13' 2023-02-25T08:45:41.301 INFO:teuthology.results:seed: '3646' 2023-02-25T08:56:07.403 INFO:root:teuthology version: 0.0.1.dev43+ge06b2e0 2023-02-25T08:56:07.407 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-02-25T10:08:12.385 INFO:teuthology.results:Tests finished! gathering results... 2023-02-25T10:08:12.954 INFO:scrape:Found 12 jobs 2023-02-25T10:08:14.158 INFO:scrape:Found 3 distinct failure reasons 2023-02-25T10:08:14.159 INFO:scrape:Failure: Test failure: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS) 2023-02-25T10:08:14.159 INFO:scrape:1 jobs: ['7186377'] 2023-02-25T10:08:14.160 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'task/test_nfs}'] 2023-02-25T10:08:14.160 INFO:scrape: 2023-02-25T10:08:14.160 INFO:scrape:Failure: Command failed on smithi035 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4867fd94-b4ec-11ed-9ae8-001a4aab830c -e sha1=ea28892ec541d00ce2ebe1cf372910d2ffd54fa1 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' 2023-02-25T10:08:14.160 INFO:scrape:2 jobs: ['7186378', '7186363'] 2023-02-25T10:08:14.161 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on'] 2023-02-25T10:08:14.161 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-02-25T10:08:14.161 INFO:scrape: 2023-02-25T10:08:14.162 INFO:scrape:Failure: {'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}, 'smithi049.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}} 2023-02-25T10:08:14.162 INFO:scrape:1 jobs: ['7186365'] 2023-02-25T10:08:14.162 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli_mon}'] 2023-02-25T10:08:14.163 INFO:scrape: 2023-02-25T10:08:14.163 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/ failed: 3 dead: 1 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7186363] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:55:18 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186363/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84e5d442ad6c434284b0d5aa9bcd75b7 Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4fdce2ca-b4eb-11ed-9ae8-001a4aab830c -e sha1=ea28892ec541d00ce2ebe1cf372910d2ffd54fa1 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' [7186377] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:31:06 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186377/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7da3a48c8e75465f98026b2d61809669 Test failure: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS) [7186378] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 01:02:14 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186378/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f2fbfe4f33d40de9a1209f0fb14a177 Command failed on smithi035 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4867fd94-b4ec-11ed-9ae8-001a4aab830c -e sha1=ea28892ec541d00ce2ebe1cf372910d2ffd54fa1 -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' Dead ================================================================= [7186365] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186365/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=460804af1d604f2a92253558db8e7ff2 {'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman- docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}, 'smithi049.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman- docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}} Pass ================================================================= [7186368] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:14 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186368/ [7186370] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:33 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186370/ [7186371] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_iscsi_pids_limit/{centos_8.stream_container_tools test_iscsi_pids_limit}} time: 00:16:10 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186371/ [7186372] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:17:13 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186372/ [7186373] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:14 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186373/ [7186374] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:44 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186374/ [7186375] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:38:41 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186375/ [7186376] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:28:50 info: http://pulpito.ceph.com/adking-2023-02-24_17:44:54-orch:cephadm-wip-adk-testing-2023-02-20-1650-distro-default-smithi/7186376/