2023-04-27T06:39:13.825 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-04-27T06:39:13.846 INFO:teuthology.results:subset: '283/1024' 2023-04-27T06:39:13.846 INFO:teuthology.results:seed: '9995' 2023-04-27T07:10:19.974 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-04-27T07:10:19.978 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-27T19:11:16.525 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-04-27T19:11:18.075 INFO:scrape:Found 22 jobs 2023-04-27T19:11:22.567 INFO:scrape:Found 5 distinct failure reasons 2023-04-27T19:11:22.568 INFO:scrape:Failure: Command failed on smithi089 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" 2023-04-27T19:11:22.568 INFO:scrape:13 jobs: ['7255613', '7255601', '7255609', '7255599', '7255598', '7255606', '7255617', '7255612', '7255607', '7255602', '7255604', '7255611', '7255600'] 2023-04-27T19:11:22.569 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-04-27T19:11:22.569 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs2', '3-final}', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-04-27T19:11:22.569 INFO:scrape: 2023-04-27T19:11:22.570 INFO:scrape:Dead: b'2023-04-27T19:11:17.630 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Apr 27 19:11:17 smithi100 ceph-da6d64ba-e4d3-11ed-9b00-001a4aab830c-mon.smithi100[123870]: cluster 2023-04-27T19:11:15.947313+0000 mgr.smithi138.jyojlw (mgr.24487) 24170 : cluster [DBG] pgmap v19361: 129 pgs: 129 active+clean; 223 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s' 2023-04-27T19:11:22.570 INFO:scrape:1 jobs: ['7255616'] 2023-04-27T19:11:22.570 INFO:scrape:suites: ['1-bootstrap/16.2.4', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-27T19:11:22.570 INFO:scrape: 2023-04-27T19:11:22.571 INFO:scrape:Dead: b'2023-04-27T18:58:30.458 DEBUG:teuthology.exit:Finished running handlers' 2023-04-27T19:11:22.571 INFO:scrape:4 jobs: ['7255603', '7255610', '7255596', '7255614'] 2023-04-27T19:11:22.571 INFO:scrape:suites intersection: ['1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-27T19:11:22.572 INFO:scrape:suites union: ['1-bootstrap/16.2.0', '1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-27T19:11:22.572 INFO:scrape: 2023-04-27T19:11:22.572 INFO:scrape:Failure: Command failed on smithi023 with status 1: 'sudo yum upgrade -y linux-firmware' 2023-04-27T19:11:22.572 INFO:scrape:1 jobs: ['7255605'] 2023-04-27T19:11:22.573 INFO:scrape:suites: ['1-bootstrap/16.2.4', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-27T19:11:22.573 INFO:scrape: 2023-04-27T19:11:22.573 INFO:scrape:Failure: {'smithi171.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-27T19:11:22.574 INFO:scrape:1 jobs: ['7255615'] 2023-04-27T19:11:22.574 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '3-final}', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-04-27T19:11:22.576 INFO:scrape: 2023-04-27T19:11:22.577 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/ failed: 14 dead: 3 running: 3 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7255598] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:07:10 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255598/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255598/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03716cee3f6f4f9fa56cd9d411f73b65 Command failed on smithi097 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255599] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:41:01 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255599/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6dee0ebafe664442bab1bbe66dbf8209 Command failed on smithi176 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255600] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:39:16 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255600/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255600/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20c835807990415cb8dec1b0a311f7be Command failed on smithi031 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255601] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255601/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255601/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c4203eea4c4042b40a68caa3922921 Command failed on smithi053 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255602] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:59 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255602/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255602/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e15e3f900354f06ad02b1acb4133b2d Command failed on smithi096 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255604] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:07:16 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255604/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255604/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01f2f007f2b84942b969c3f1a7e8ad2b Command failed on smithi073 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255605] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255605/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255605/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfeaefd1545649b299680ec4781316fc Command failed on smithi023 with status 1: 'sudo yum upgrade -y linux- firmware' [7255606] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:13:21 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255606/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255606/ sentry: https://sentry.ceph.com/organizations/ceph/?query=186e04948eea48b28ad5b9dd7099c899 Command failed on smithi039 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255607] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255607/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8aaf623428034446a8a8def24ae86998 Command failed on smithi103 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255609] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:07:16 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255609/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255609/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb5c69ea40b4430b96110c439ba20c07 Command failed on smithi097 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255611] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:39:03 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255611/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255611/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5be7a0361ac6449d93eb0c8f6c4a8c8f Command failed on smithi063 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255612] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:41:12 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255612/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255612/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcff74aeb8a24bd99aaf0b47a5e20d41 Command failed on smithi012 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255613] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255613/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255613/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09c0b7021c9d4f26a5ad16f9bbf35253 Command failed on smithi089 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7255617] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:39:03 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255617/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=404d760dd22c4585bafe32466cd5cfe1 Command failed on smithi033 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" Dead ================================================================= [7255596] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255596/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255596/ hit max job timeout [7255603] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255603/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255603/ hit max job timeout [7255615] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:36:15 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255615/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255615/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e60b68053c76425da27d4be70256c05f {'smithi171.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Running ================================================================= [7255610] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255610/ [7255614] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255614/ [7255616] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255616/ Pass ================================================================= [7255597] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_nfs} time: 00:36:52 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255597/ [7255608] orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_nfs} time: 00:44:42 info: http://pulpito.ceph.com/vshankar-2023-04-27_06:36:49-orch-wip-vshankar-testing-20230427.052001-testing-default-smithi/7255608/