2023-08-30T04:48:51.201 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-30T04:48:51.202 INFO:teuthology.results:subset: '344/1024' 2023-08-30T04:48:51.203 INFO:teuthology.results:seed: '8988' 2023-08-30T05:29:22.259 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-30T05:29:22.263 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-30T06:26:27.006 INFO:teuthology.results:Tests finished! gathering results... 2023-08-30T06:26:32.947 INFO:scrape:Found 75 jobs 2023-08-30T06:26:36.323 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383726/teuthology.log 2023-08-30T06:26:36.662 INFO:scrape:Found 3 distinct failure reasons 2023-08-30T06:26:36.663 INFO:scrape:Failure: Command failed on smithi133 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" 2023-08-30T06:26:36.663 INFO:scrape:36 jobs 2023-08-30T06:26:36.664 INFO:scrape:suites intersection: ['0-nvme-loop'] 2023-08-30T06:26:36.664 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '3-final}', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'start}'] 2023-08-30T06:26:36.665 INFO:scrape: 2023-08-30T06:26:36.665 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-08-30T06:26:36.665 INFO:scrape:1 jobs: ['7383761'] 2023-08-30T06:26:36.666 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'objectstore/bluestore-bitmap', 'orch:cephadm/nfs/{begin/{0-install', 'overrides/ignorelist_health', 'supported-random-distros$/{rhel_8}', 'tasks/nfs}'] 2023-08-30T06:26:36.667 INFO:scrape: 2023-08-30T06:26:36.667 INFO:scrape:Dead: None 2023-08-30T06:26:36.667 INFO:scrape:1 jobs: ['7383726'] 2023-08-30T06:26:36.668 INFO:scrape: 2023-08-30T06:26:36.675 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/ failed: 37 dead: 1 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7383716] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:14:31 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383716/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383716/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e2004b2c0dc456da7295d98368ac3bc Command failed on smithi124 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383719] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383719/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383719/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9f8a97210e749dfaeb517c4a191c17c Command failed on smithi045 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383720] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:59 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383720/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b893cc9b0a8d4ac288c60dd507c6f784 Command failed on smithi139 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383722] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:08:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383722/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c40b9e74a27c4199b4b499eb31972ba2 Command failed on smithi129 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383723] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:14:05 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383723/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fc04624b9df45f1851ea517d448976c Command failed on smithi002 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383725] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:10:49 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383725/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00e42eb4948e49b880feafbe120dc6c6 Command failed on smithi012 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383727] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:46 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383727/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a65ab9430e7d4263ba9cd46003ef4436 Command failed on smithi102 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383732] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383732/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3364095e00f49cdb6745674e6a5cb40 Command failed on smithi141 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383733] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383733/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383733/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bd67a2f42e14530b79638afffddc27d Command failed on smithi133 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383734] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:13:04 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383734/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a94b96c3406c48f692a482b4fa9daec9 Command failed on smithi094 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383736] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383736/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8827df2ade3848dd880b933ee92c8ff2 Command failed on smithi089 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383737] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383737/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383737/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abd8705687ac4c44abdaa4da4c9add2a Command failed on smithi120 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383741] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:07:59 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383741/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c70ae3760d90467bbe828d98108cb498 Command failed on smithi046 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383743] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:10:58 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383743/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e785d099f4dd4c39b66e0e1b171c6608 Command failed on smithi059 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383744] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383744/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06f58599e215485db8553186be6ff1ba Command failed on smithi140 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383745] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:10:59 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383745/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6156f608545842b089693545d75c266a Command failed on smithi064 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383748] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:07:58 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383748/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=864567111c524f918e1bb281091bc239 Command failed on smithi083 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383749] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383749/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4edac5150ab24a518e017ce2c0745e3c Command failed on smithi142 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383753] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:13:32 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383753/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=226e5ad1749c48938bcde50d8e163bac Command failed on smithi072 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383754] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:13:40 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383754/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0f931310f2740a58f8f86ae65c30288 Command failed on smithi050 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383756] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:13:15 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383756/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3082ae9c237e40f2ad0751999de24f89 Command failed on smithi173 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383757] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:07:56 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383757/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f90cf0485d4448bc81c58858ab90914e Command failed on smithi130 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383758] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383758/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e1710d10aa343d7865cc89289481edc Command failed on smithi093 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383761] orch:cephadm/nfs/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/nfs} ----------------------------------------------------------------- time: 00:22:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383761/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ff06a7e1f3a48c0996bba06ef4cebbc Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7383764] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:10:52 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383764/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c125d13a9c284917b803f03e1361bc69 Command failed on smithi124 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383766] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:08 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383766/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63e288cd54bf419c955c0443aeef82ed Command failed on smithi012 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383769] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:20 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383769/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1761ccf79ba34ead81f7f5fa8db5ca26 Command failed on smithi088 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383770] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383770/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ab2d0520a4d4134a8189e583b429a0c Command failed on smithi017 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383772] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:15:28 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383772/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383772/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ee8b58c855d400687813c2637f66805 Command failed on smithi016 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383773] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383773/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=814b21e3e3cb4fc4840f5572917a2a85 Command failed on smithi029 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383776] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:14:37 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383776/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d858bdbb05da45cdbeb76a9e1ff9eca0 Command failed on smithi133 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383780] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383780/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=871c05fa83b84b988a673e38eea51cf7 Command failed on smithi120 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383781] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383781/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d03d6925082a429ab61e0bbcd98fb444 Command failed on smithi159 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383783] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383783/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e5f6a7963ce437cb10e859e2c12218b Command failed on smithi046 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383784] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:10:53 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383784/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d31370c55164ecf817cf7e71c1ca1b5 Command failed on smithi078 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383785] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:15:07 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383785/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383785/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3776ea8f410944bbb525b355fd595006 Command failed on smithi122 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7383786] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:10:57 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383786/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88889ddbe448471eab413346f885b48d Command failed on smithi059 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" Dead ================================================================= [7383726] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383726/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383726/ Error reimaging machines: Timeout opening channel. Pass ================================================================= [7383713] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:45:09 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383713/ [7383714] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:47:46 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383714/ [7383715] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:22:21 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383715/ [7383717] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:48:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383717/ [7383718] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:42 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383718/ [7383721] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption} time: 00:17:09 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383721/ [7383724] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} time: 00:22:02 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383724/ [7383728] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:41:15 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383728/ [7383729] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:33:17 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383729/ [7383730] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:27:53 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383730/ [7383731] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_ca_signed_key} time: 00:23:13 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383731/ [7383735] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm} time: 00:24:34 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383735/ [7383738] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:44:40 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383738/ [7383739] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:41:04 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383739/ [7383740] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:30:21 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383740/ [7383742] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos} time: 00:09:21 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383742/ [7383746] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:14 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383746/ [7383747] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:21:13 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383747/ [7383750] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 01:04:10 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383750/ [7383751] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:58:05 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383751/ [7383752] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:48:48 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383752/ [7383755] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:21:06 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383755/ [7383759] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_orch_cli} time: 00:28:02 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383759/ [7383760] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:44:37 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383760/ [7383762] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli} time: 00:28:12 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383762/ [7383763] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 00:29:32 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383763/ [7383765] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final} time: 00:16:23 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383765/ [7383767] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:44:25 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383767/ [7383768] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:32:00 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383768/ [7383771] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:39:35 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383771/ [7383774] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:33:13 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383774/ [7383775] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:53 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383775/ [7383777] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:22:29 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383777/ [7383778] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:53:32 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383778/ [7383779] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:41:57 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383779/ [7383782] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:23:20 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383782/ [7383787] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:07 info: http://pulpito.ceph.com/vshankar-2023-08-30_04:48:33-orch:cephadm-main-testing-default-smithi/7383787/