2023-04-22T21:20:34.691 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-22T21:20:34.692 INFO:teuthology.results:subset: '1/13' 2023-04-22T21:20:34.692 INFO:teuthology.results:seed: '4058' 2023-04-22T21:38:04.833 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-22T21:38:04.870 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-22T22:59:11.353 INFO:teuthology.results:Tests finished! gathering results... 2023-04-22T22:59:11.876 INFO:scrape:Found 12 jobs 2023-04-22T22:59:13.193 INFO:scrape:Found 4 distinct failure reasons 2023-04-22T22:59:13.194 INFO:scrape:Failure: reached maximum tries (300) after waiting for 300 seconds 2023-04-22T22:59:13.194 INFO:scrape:1 jobs: ['7248621'] 2023-04-22T22:59:13.194 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] 2023-04-22T22:59:13.195 INFO:scrape: 2023-04-22T22:59:13.195 INFO:scrape:Failure: Command failed on smithi047 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.47:/fake /mnt/foo'" 2023-04-22T22:59:13.195 INFO:scrape:2 jobs: ['7248626', '7248618'] 2023-04-22T22:59:13.196 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}'] 2023-04-22T22:59:13.196 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-04-22T22:59:13.196 INFO:scrape: 2023-04-22T22:59:13.197 INFO:scrape:Failure: Command failed on smithi150 with status 2: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6a217d2-e157-11ed-9b00-001a4aab830c -e sha1=797ed80f1a302a44ea70f221af60167435b6b296 -- bash -c \'ceph orch ps --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] | select(.daemon_type=="mgr") | select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\'' 2023-04-22T22:59:13.197 INFO:scrape:2 jobs: ['7248628', '7248619'] 2023-04-22T22:59:13.197 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on'] 2023-04-22T22:59:13.198 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-04-22T22:59:13.198 INFO:scrape: 2023-04-22T22:59:13.198 INFO:scrape:Failure: {'smithi174.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d03f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}, 'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-22T22:59:13.199 INFO:scrape:2 jobs: ['7248627', '7248622'] 2023-04-22T22:59:13.199 INFO:scrape:suites intersection: ['agent/on'] 2023-04-22T22:59:13.199 INFO:scrape:suites union: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'start}', 'task/test_orch_cli_mon}'] 2023-04-22T22:59:13.200 INFO:scrape: 2023-04-22T22:59:13.200 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/ failed: 5 dead: 2 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7248618] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:11 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248618/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a5e689e0d75421bbf12a158c70b277e Command failed on smithi043 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.43:/fake /mnt/foo'" [7248619] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:53:21 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248619/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248619/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cce2db13776249698da5d8f4a21529fb Command failed on smithi017 with status 2: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3927e1c-e15d-11ed-9b00-001a4aab830c -e sha1=797ed80f1a302a44ea70f221af60167435b6b296 -- bash -c \'ceph orch ps --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] | select(.daemon_type=="mgr") | select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\'' [7248621] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 01:02:08 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248621/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f51278d18d14335ba13252cfbc0f30e reached maximum tries (300) after waiting for 300 seconds [7248626] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:31 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248626/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248626/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fca0398bf89b49b082289b28e2310e53 Command failed on smithi047 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.47:/fake /mnt/foo'" [7248628] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:24:20 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248628/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd1826cfa05f432b89f6999671330305 Command failed on smithi150 with status 2: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6a217d2-e157-11ed-9b00-001a4aab830c -e sha1=797ed80f1a302a44ea70f221af60167435b6b296 -- bash -c \'ceph orch ps --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] | select(.daemon_type=="mgr") | select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\'' Dead ================================================================= [7248622] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:38:35 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248622/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=785d410303664e8980a74a230c724efe {'smithi023.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d0 3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} [7248627] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:40:48 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248627/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=237933066de142d9866faf9d126995a1 {'smithi174.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d0 3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}, 'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12 af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Pass ================================================================= [7248620] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:51:17 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248620/ [7248623] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 01:00:27 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248623/ [7248624] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:50:57 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248624/ [7248625] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:46:19 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248625/ [7248629] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:47:49 info: http://pulpito.ceph.com/adking-2023-04-22_17:05:01-orch:cephadm-wip-adk-testing-2023-04-20-0105-distro-default-smithi/7248629/