2023-05-02T04:18:04.066 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-02T04:18:04.067 INFO:teuthology.results:subset: '503/1024' 2023-05-02T04:18:04.068 INFO:teuthology.results:seed: '9500' 2023-05-02T04:42:58.411 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-02T04:42:58.415 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-02T05:42:03.137 INFO:teuthology.results:Tests finished! gathering results... 2023-05-02T05:42:04.665 INFO:scrape:Found 22 jobs 2023-05-02T05:42:05.946 INFO:scrape:Found 1 distinct failure reasons 2023-05-02T05:42:05.947 INFO:scrape:Failure: Command failed on smithi089 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" 2023-05-02T05:42:05.947 INFO:scrape:14 jobs: ['7260312', '7260315', '7260319', '7260325', '7260311', '7260321', '7260320', '7260327', '7260318', '7260331', '7260326', '7260328', '7260330', '7260317'] 2023-05-02T05:42:05.948 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-05-02T05:42:05.948 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs2', '3-final}', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-05-02T05:42:05.948 INFO:scrape: 2023-05-02T05:42:05.950 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7260311] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:05 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260311/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260311/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e768154117248369cd1b26238bdc0f7 Command failed on smithi017 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260312] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260312/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5945ff4fac8542cf94885abecf3b8b5b Command failed on smithi089 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260315] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:10:05 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260315/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260315/ sentry: https://sentry.ceph.com/organizations/ceph/?query=740b06592dff4e42b5b3ea3dec4fcdbc Command failed on smithi023 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260317] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:10:09 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260317/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0aae3359259d448eb7b6e30ed8c9faac Command failed on smithi115 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260318] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:13:32 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260318/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260318/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf52637cd5354919a4417f7ad8aeb969 Command failed on smithi161 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260319] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260319/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35b889df23154055bd8cc62f08177cfb Command failed on smithi153 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260320] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:07:18 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260320/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260320/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d69ed597c964f39b65383a3c3da48bd Command failed on smithi016 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260321] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:10:11 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260321/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=609cd0a720f7490dbfa77dc1ece649fc Command failed on smithi006 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260325] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260325/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28f062dacbff408b97ed38b70409449c Command failed on smithi134 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260326] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:07:14 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260326/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad22fec538574a71957af471c10e28b5 Command failed on smithi137 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260327] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260327/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7170b443041a4d4a8a058f57f32a458e Command failed on smithi100 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260328] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:09:55 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260328/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb9a7dd8f323420ab7d9f6480821f84c Command failed on smithi057 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260330] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:11:42 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260330/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260330/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dec3286d9db549669ad3dd9875345f95 Command failed on smithi012 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" [7260331] orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:12:01 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260331/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260331/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a98f09d4a034de3ac14d97715099619 Command failed on smithi002 with status 1: "grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype" Pass ================================================================= [7260313] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:47:40 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260313/ [7260314] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_nfs} time: 00:40:30 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260314/ [7260316] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:48:45 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260316/ [7260322] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:46:18 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260322/ [7260323] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:52:11 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260323/ [7260324] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_nfs} time: 00:36:49 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260324/ [7260329] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:52:53 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260329/ [7260332] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:46:51 info: http://pulpito.ceph.com/vshankar-2023-05-02_04:17:08-orch-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260332/